Image-capturing device and image processing device

ABSTRACT

An image-capturing device includes: an image sensor that includes an imagecapturing area where an image of a subject is captured; a setting unit that sets imagecapturing conditions to be applied to the image-capturing area; a selection unit that selects pixels to be used for interpolation from pixels present in the image-capturing area; and a generation unit that generates an image of the subject captured in the image-capturing area with signals generated through interpolation executed by using signals output from the pixels selected by the selection unit, wherein: the selection unit makes a change in selection of at least some of the pixels to be selected depending upon the image-capturing conditions set by the setting unit.

This application is a continuation application of U.S. application Ser.No. 15/764,917, filed Mar. 30, 2018, which is in turn a National StageApplication of PCT/JP2016/078313, filed Sep. 26, 2016, which itselfclaims priority to Japanese Application No. 2015-195288, filed Sep. 30,2015.

TECHNICAL FIELD

The present invention relates to an image-capturing device and an imageprocessing device.

BACKGROUND ART

There is an image-capturing device equipped with an image sensor thatallows an image-capturing condition different from the image-capturingcondition selected for another area to be set for an area in the imagingfield is known in the related art (see PTL1). However, there is an issueto be addressed in that image data obtained over areas with differentimage-capturing conditions applied thereto cannot be handled in a mannersimilar to that with which image data obtained in areas with matchingimage-capturing conditions applied thereto are used.

CITATION LIST Patent Literature

PTL1: Japanese Laid Open Patent Publication No. 2006-197192

SUMMARY OF INVENTION

According to the 1st aspect of the present invention, an image-capturingdevice comprises: an image sensor that includes an image-capturing areawhere an image of a subject is captured; a setting unit that setsimage-capturing conditions to be applied to the image-capturing area; aselection unit that selects pixels to be used for interpolation frompixels present in the image-capturing area; and a generation unit thatgenerates an image of the subject captured in the image-capturing areawith signals generated through interpolation executed by using signalsoutput from the pixels selected by the selection unit, wherein: theselection unit makes a change in selection of at least some of thepixels to be selected depending upon the image-capturing conditions setby the setting unit.

According to the 15th aspect of the present invention, animage-capturing device comprises: an image sensor that includes a firstimage-capturing area set so as to capture an image of a subject underfirst image-capturing conditions, a second image-capturing area set soas to capture an image of the subject under second image-capturingconditions different from the first image-capturing conditions and athird image-capturing area set so as to capture an image of the subjectunder third image-capturing conditions different from the secondimage-capturing conditions; a selection unit that selects pixels, to beused for purposes of interpolation executed for a pixel present in thefirst image-capturing area among pixels present in the secondimage-capturing area, and pixels present in the third image-capturingarea; and a generation unit that generates an image of the subjectcaptured in the first image-capturing area with signals generatedthrough interpolation executed by using signals output from the pixelsselected by the selection unit.

According to the 16th aspect of the present invention, animage-capturing device comprises: an image sensor that includes a firstimage-capturing area set so as to capture an image of a subject underfirst image-capturing conditions and a second image-capturing area setso as to capture an image of the subject under second image-capturingconditions different from the first image-capturing conditions; aselection unit that selects pixels to be used for purposes ofinterpolation executed for a pixel present in the first image-capturingarea among pixels present in the first image-capturing area and pixelspresent in the second image-capturing area; and a generation unit thatgenerates an image of the subject captured in the first image-capturingarea with signals generated through interpolation executed by usingsignals output from the pixels selected by the selection unit.

According to the 17th aspect of the present invention, animage-capturing device comprises: an image sensor that includes a firstimage-capturing area where an image of a subject is captured, a secondimage-capturing area where an image of the subject is captured and athird image-capturing area where an image of the subject is captured; asetting unit that sets first image-capturing conditions asimage-capturing conditions for the first image-capturing area, setssecond image-capturing conditions, different from the firstimage-capturing conditions, as image-capturing conditions for the secondimage-capturing area and sets third image-capturing conditions,different from the first image-capturing conditions by an extent smallerthan an extent of difference between the first image-capturingconditions and the second image-capturing conditions, as image-capturingconditions for the third image-capturing area; a selection unit thatselects pixels to be used for purposes of interpolation executed for apixel present in the first image-capturing area among pixels present inthe first image-capturing area, pixels present in the secondimage-capturing area and pixels present in the third image-capturingarea; and a generation unit that generates an image of the subjectcaptured in the first image-capturing area with signals generatedthrough interpolation executed by using signals output from the pixelsselected by the selection unit.

According to the 18th aspect of the present invention, animage-capturing device comprises: an image sensor that includes a firstimage-capturing area where an image of a subject is captured, a secondimage-capturing area where an image of the subject is captured and athird image-capturing area where an image of the subject is captured,set apart from the first image-capturing area by a distance greater thana distance between the first image-capturing area and the secondimage-capturing area; a setting unit that sets image-capturingconditions different from image-capturing conditions selected for thefirst image-capturing area, as image-capturing conditions for the secondimage-capturing area; a selection unit that selects pixels to be usedfor purposes of interpolation executed for a pixel present in the firstimage-capturing area among pixels present in the first image-capturingarea, pixels present in the second image-capturing area and pixelspresent in the third image-capturing area; and a generation unit thatgenerates an image of the subject captured in the first image-capturingarea with signals generated through interpolation executed by usingsignals output from the pixels selected by the selection unit.

According to the 19th aspect of the present invention, animage-capturing device comprises: an image sensor that includes animage-capturing area where an image of a subject is captured; a settingunit that sets image-capturing conditions to be applied to theimage-capturing area; and a generation unit that generates an image ofthe subject captured in the image-capturing area with signals generatedthrough interpolation executed by using signals output from pixelsselected as pixels to be used for purposes of interpolation and presentin the image-capturing area, wherein: a change is made in selection ofat least some of the pixels to be selected depending upon theimage-capturing conditions set by the setting unit.

According to the 20th aspect of the present invention, animage-capturing device comprises: an image sensor that includes a firstimage-capturing area set so as to capture an image of a subject underfirst image-capturing conditions and a second image-capturing area setso as to capture an image of the subject under second image-capturingconditions different from the first image-capturing conditions; and ageneration unit that generates an image of the subject captured in thefirst image-capturing area with signals generated through interpolationexecuted by using signals output from pixels, selected as pixels to beused for purposes of interpolation executed for a pixel present in thefirst image-capturing area, among pixels present in the firstimage-capturing area and pixels present in the second image-capturingarea.

According to the 21st aspect of the present invention, animage-capturing device comprises: an image sensor that includes animage-capturing area where an image of a subject is captured; a settingunit that sets image-capturing conditions to be applied to theimage-capturing area; and a generation unit that generates an image ofthe subject captured in the image-capturing area with signals generatedthrough noise reduction executed by using signals output from pixelsthat output noise reduction signals, selected from pixels present in theimage-capturing area, wherein: a change is made in selection of at leastsome of the pixels to be selected depending upon the image-capturingconditions set by the setting unit.

According to the 22nd aspect of the present invention, animage-capturing device comprises: an image sensor that includes a firstimage-capturing area set so as to capture an image of a subject underfirst image-capturing conditions, a second image-capturing area set soas to capture an image of the subject under second image-capturingconditions different from the first image-capturing conditions and athird image-capturing area set so as to capture an image of the subjectunder third image-capturing conditions different from the secondimage-capturing conditions; a selection unit that selects pixels to beused to reduce noise for a pixel present in the first image-capturingarea among pixels present in the second image-capturing area and pixelspresent in the third image-capturing area; and a generation unit thatgenerates an image of the subject captured in the first image-capturingarea with signals having undergone noise reduction executed by usingsignals output from the pixels selected from the pixels present in thesecond image-capturing area and the pixels present in the thirdimage-capturing area, as pixels to output signals to be used to reducenoise in a signal at a pixel present in the first image-capturing area.

According to the 23rd aspect of the present invention, animage-capturing device comprises: an image sensor that includes a firstimage-capturing area set so as to capture an image of a subject underfirst image-capturing conditions and a second image-capturing area setso as to capture an image of the subject under second image-capturingconditions different from the first image-capturing conditions; and ageneration unit that generates an image of the subject captured in thefirst image-capturing area with signals generated through interpolationexecuted by using signals output from pixels selected from pixelspresent in the first image-capturing area and pixels present in thesecond image-capturing area, as pixels to output signals to be used toreduce noise for a pixel present in the first image-capturing area.

According to the 24th aspect of the present invention, animage-capturing device comprises: an image sensor that includes animage-capturing area where an image of a subject is captured; a settingunit that sets image-capturing conditions to be applied to theimage-capturing area; and a generation unit that generates an image ofthe subject captured in the image-capturing area with signals havingundergone image processing executed by using signals output from pixelsselected as pixels to be used in image processing, wherein: a change ismade in selection of at least some of the pixels to be selecteddepending upon the image-capturing conditions set by the setting unit.

According to the 25th aspect of the present invention, an imageprocessing device comprises: a selection unit that selects signals to beused for purposes of interpolation, among signals output from pixelspresent in an image-capturing area of an image sensor; and a generationunit that generates an image of a subject captured in theimage-capturing area with signals generated through interpolationexecuted by using the signals selected by the selection unit, wherein:the selection unit makes a change in selection of at least some of thepixels to be selected depending upon the image-capturing conditions setfor the image-capturing area.

According to the 26th aspect of the present invention, an imageprocessing device comprises: a selection unit that selects signals to beused for noise reduction, among signals output among pixels present inan image-capturing area of an image sensor; and a generation unit thatgenerates an image of a subject captured in the image-capturing areawith signals having undergone noise reduction executed by using thesignals selected by the selection unit, wherein: the selection unitmakes a change in selection of at least some of the pixels to beselected depending upon the image-capturing conditions set for theimage-capturing area.

According to the 27th aspect of the present invention, an imageprocessing device comprises: a selection unit that selects signals to beused for purposes of interpolation, among signals output from pixelspresent in an image-capturing area of an image sensor; and a displayunit that brings up on display an image of a subject captured in theimage-capturing area, generated with signals generated throughinterpolation executed by using the signals selected by the selectionunit, wherein: the selection unit makes a change in selection of atleast some of the pixels to be selected depending upon theimage-capturing conditions set for the image-capturing area.

According to the 28th aspect of the present invention, an imageprocessing device comprises: a selection unit that selects signals to beused for noise reduction, among signals output from pixels present in animage-capturing area of an image sensor; and a display unit that bringsup on display an image of a subject captured in the image-capturingarea, generated with signals having undergone noise reduction executedby using the signals selected by the selection unit, wherein: theselection unit makes a change in selection of at least some of thepixels to be selected depending upon the image-capturing conditions setfor the image-capturing area.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A block diagram presenting an example of a structure that may beadopted in the camera achieved in a first embodiment

FIG. 2 A sectional view of a stacked image sensor

FIG. 3 An illustration of the pixel arrangement and unit areas at theimage-capturing chip

FIG. 4 An illustration of the circuits in a unit area

FIG. 5 A schematic illustration of an image of a subject formed at theimage sensor in the camera

FIG. 6 An example of an image-capturing condition setting screen

FIG. 7 An example of an area around the boundary of a first area in alive view image illustrated in FIG. 7(a), an enlarged view of the areaaround the boundary presented in FIG. 7(b), an enlarged view of a targetpixel and reference pixels presented in FIG. 7(c), and an enlarged viewof a target pixel and reference pixels designated in a second embodimentpresented in FIG. 7(d)

FIG. 8 An example of an arrangement of photoelectric conversion signalsoutput from pixels illustrated in FIG. 8(a), a diagram illustrating howG-color component image data are generated through interpolationpresented in FIG. 8(b), and an example of G-color component image dataresulting from the interpolation illustrated in FIG. 8(c)

FIG. 9 A diagram showing only the R-color component image data in FIG.8(a), presented in FIG. 9(a), an illustration of interpolation for thechrominance component Cr presented in FIG. 9(b), and a diagramillustrating how image data for the chrominance component Cr aregenerated through interpolation presented in FIG. 9(c)

FIG. 10 A diagram showing only the B-color component image data in FIG.8(a), presented in FIG. 10(a), an illustration of interpolation for thechrominance component Cb presented in FIG. 10(b), and a diagramillustrating how image data for the chrominance component Cb aregenerated through interpolation presented in FIG. 10(c)

FIG. 11 A diagram presenting examples of positions at which focusdetection pixels may be disposed at the image-capturing surface

FIG. 12 An enlarged view of an area of a focus detection pixel line

FIG. 13 An enlarged view of the focus detection position

FIG. 14 An example of a template image representing a detection targetobject illustrated in FIG. 14(a), and an example of a live view imageand the corresponding search range presented in FIG. 14(b)

FIG. 15 A flowchart of the image-capturing processing executed bysetting different image-capturing conditions for different areas

FIG. 16 Examples of positions that may be taken for a first area and asecond area at the image-capturing surface of the image sensor,presented in FIG. 16(a) through FIG. 16(c)

FIG. 17 A block diagram presenting an example of a structure that may beadopted in the image-capturing system achieved in Variation 8

FIG. 18 An illustration showing how a program is provided a mobiledevice

FIG. 19 A block diagram presenting an example of a structure that may beadopted in the camera achieved in a third embodiment

FIG. 20 A schematic diagram indicating the correspondence between theindividual blocks and a plurality of selection units

FIG. 21 A sectional view of a stacked image sensor

FIG. 22 A schematic illustration showing how the first image data andthe second image data are processed in relation to image processing

FIG. 23 A schematic illustration showing how the first image data andthe second image data are processed in relation to focus detectionprocessing

FIG. 24 A schematic illustration showing how the first image data andthe second image data are processed in relation to subject detectionprocessing

FIG. 25 A schematic illustration showing how the first image data andthe second image data are processed when the image-capturing conditionsare set through, for instance, exposure calculation processing

FIG. 26 A schematic illustration showing how the first image data andthe second image data are processed in Variation 10

FIG. 27 A schematic illustration showing how the first image data andthe second image data are processed in Variation 10

FIG. 28 A schematic illustration showing how the first image data andthe second image data are processed in Variation 10

FIG. 29 A schematic illustration showing how the first image data andthe second image data are processed in Variation 10

DESCRIPTION OF EMBODIMENTS

The image processing device achieved in an embodiment may be installedin an electronic device such as a digital camera, as described below. Acamera 1 (see FIG. 1) is configured so as to enable image-capturingoperation to be executed under different conditions set for differentareas of the image-capturing surface at an image sensor 32 a. An imageprocessing unit 33 executes processing optimal for each of the areas forwhich different image-capturing conditions have been selected. Such acamera 1 will be described in detail in reference to drawings.

Description of the Camera First Embodiment

FIG. 1 is a block diagram presenting an example of a structure that maybe adopted in the camera 1 achieved in the first embodiment. The camera1 in FIG. 1 comprises an image-capturing optical system 31, animage-capturing unit 32, the image processing unit 33, a control unit34, a display unit 35, an operation member 36 and a recording unit 37.

The image-capturing optical system 31 guides a light flux departing asubject field to the image-capturing unit 32. At the image-capturingunit 32, which includes the image sensor 32 a and a drive unit 32 b, animage of the subject, formed via the image-capturing optical system 31,undergoes photoelectric conversion. The image-capturing unit 32 iscapable of capturing an image under uniform conditions set for theentire range of the image-capturing surface at the image sensor 32 a orcapturing an image under different conditions for different areas of theimage-capturing surface at the image sensor 32 a. The image-capturingunit 32 will be described in detail later. The drive unit 32 b generatesa drive signal needed to enable accumulation control at the image sensor32 a. An image-capturing instruction indicating an electric chargeaccumulation time and the like for the image-capturing unit 32 istransmitted from the control unit 34 to the drive unit 32 b.

The image processing unit 33 includes an input unit 33 a, a selectionunit 33 b and a generation unit 33 c. Image data obtained by theimage-capturing unit 32 are input to the input unit 33 a. The selectionunit 33 b executes preliminary processing on the image data input asdescribed above. The preliminary processing will be described in detaillater. The generation unit 33 c generates an image based upon the inputimage data mentioned above and image data resulting from the preliminaryprocessing. In addition, the generation unit 33 c executes imageprocessing on the image data. This image processing includes, forinstance, color interpolation processing, defective pixel correctionprocessing, edge enhancement processing, noise reduction processing,white balance adjustment processing, gamma correction processing,display luminance adjustment processing and saturation adjustmentprocessing. In addition, the generation unit 33 c generates an image tobe brought up on display at the display unit 35.

The control unit 34, which may be constituted with, for instance, a CPU,controls the overall operation in the camera 1. For instance, thecontrol unit 34 executes a predetermined exposure calculation based uponphotoelectric conversion signals obtained via the image-capturing unit32, determines exposure conditions such as the electric chargeaccumulation time (exposure time), the aperture value to be set for theimage-capturing optical system 31 and the ISO sensitivity, which must beset for the image sensor 32 a to achieve the optimal exposure, andissues an instruction for the drive unit 32 b accordingly. In addition,it determines image processing conditions under which the saturation(chroma), the contrast, the sharpness and the like are to be adjusted,in correspondence to the image-capturing scene mode currently set in thecamera 1 and the types of subject elements having been detected. It thenissues an instruction to the image processing unit 33 accordingly.Detection of subject elements will be explained later.

The control unit 34 includes an object detection unit 34 a, a settingunit 34 b, an image-capturing control unit 34 c and an AF operation unit34 d. While these units are achieved in software with the control unit34 executing a program installed in a nonvolatile memory (not shown),they may instead be configured with an ASIC or the like instead.

The object detection unit 34 a executes object recognition processing ofthe known art so as to detect subject elements, such as a person (aperson's face), an animal such as a dog or a cat (an animal face), aplant, a vehicle such as a bicycle, an automobile or a train, abuilding, a still object, a landscape such as mountains or clouds, and adetermined specific object, in an image obtained via the image-capturingunit 32. The setting unit 34 b divides the imaging field at theimage-capturing unit 32 into a plurality of areas that contain subjectelements detected as described above.

The setting unit 34 b also sets image-capturing conditions for theplurality of areas. The image-capturing conditions include the exposureconditions (the electric charge accumulation time, the gain, the ISOsensitivity, the frame rate and the like) explained earlier and theimage processing conditions (e.g., a white balance adjustment parameter,a gamma correction curve, a display brightness adjustment parameter, asaturation adjustment parameter and the like). It is to be noted thatuniform image-capturing conditions may be set for all of the pluralityof areas or different image-capturing conditions may be set fordifferent areas among the plurality of areas.

The image-capturing control unit 34 c controls the image-capturing unit32 (image sensor 32 a) and the image processing unit 33 by adopting theimage-capturing conditions set by the setting unit 34 b for theindividual areas. Thus, the image-capturing unit 32 can be engaged inimage-capturing operation under different exposure conditions set fordifferent areas among the plurality of areas and the image processingunit 33 can be engaged in image processing under different imageprocessing conditions set for different areas among the plurality ofareas. There are no restrictions imposed with regard to the number ofpixels making up an area. For instance, each area may be made up with1000 pixels or a single pixel. In addition, the number of pixels makingup a given area may be different from the number of pixels making upanother area.

The AF operation unit 34 d controls autofocus adjustment (autofocus: AF)operation through which focus is adjusted at a specific position withinthe imaging field (hereafter referred to as a focus detection position)for the subject corresponding to the specific position. Based uponarithmetic operation results, the AF operation unit 34 d transmits adrive signal based upon which a focus lens in the image-capturingoptical system 31 is to be driven to an in-focus position, to the driveunit 32 b. The processing executed by the AF operation unit 34 d forautofocus adjustment is also referred to as focus detection processing.The focus detection processing will be described in detail later.

An image generated via the image processing unit 33, an image resultingfrom the image processing executed at the image processing unit 33, animage read out from the recording unit 37, or the like is reproduced andbrought up on display at the display unit 35. At the display unit 35, anoperation menu screen, a setting screen in which image-capturingconditions are set, and the like are also brought up on display.

The operation member 36 is configured with various operation numberssuch as a shutter release button and a menu button. The operation member36 outputs an operation signal corresponding to a specific operationamong various operations to the control unit 34. The operation member 36includes a touch operation member disposed at the display surface of thedisplay unit 35.

In response to an instruction issued by the control unit 34, therecording unit 37 records image data and the like into a recordingmedium constituted with, for instance, a memory card (not shown). Therecording unit 37 also reads out image data recorded in the recordingmedium in response to an instruction issued by the control unit 34.

<Description of a Stacked Image Sensor>

A stacked image sensor 100 representing an example of the image sensor32 a described above will be explained next. FIG. 2 is a sectional viewof the image sensor 100. The image sensor 100 includes animage-capturing chip 111, a signal processing chip 112 and a memory chip113. The image-capturing chip 111 is stacked on the signal processingchip 112. The signal processing chip 112 is stacked on the memory chip113. Electrical connections are achieved via connecting portions 109 forthe image-capturing chip 111 and the signal processing chip 112 and alsofor the signal processing chip 112 and the memory chip 113. Theconnecting portions 109 may be, for instance, bumps or electrodes. Theimage-capturing chip 111 captures an image formed with light departing asubject and generates image data. The image-capturing chip 111 outputsthe image data to the signal processing chip 112. At the signalprocessing chip 112, signal processing is executed on the image dataoutput from the image-capturing chip 111. In the memory chip 113, whichincludes a plurality of memories, the image data are stored. It is to benoted that the image sensor 100 may be configured with animage-capturing chip and a signal processing chip only. In the imagesensor 100 configured with an image-capturing chip and a signalprocessing chip, a storage unit where image data are stored may beincluded in the signal processing chip, or a separate storage unitindependent of the image sensor 100 may be used for data storage.

As shown in FIG. 2, incident light enters the image sensor primarilytoward a Z axis+side indicated by the unfilled arrow. In addition, thedirection running to the left on the drawing sheet and perpendicular tothe Z axis will be referred to as an X axis+direction and a directionrunning toward the viewer of the drawing away from the drawing sheet,perpendicular to the Z axis and the X axis, will be referred to as a Yaxis+direction with regard to the coordinate axes. In some of thedrawings to be referred to in the subsequent description, coordinateaxes will be presented so as to indicate the orientations of theindividual drawings in reference to the coordinate axes in FIG. 2.

The image-capturing chip 111 may be, for instance, a CMOS image sensor.More specifically, the image-capturing chip 111 is a backsideillumination CMOS image sensor. The image-capturing chip 111 includes amicrolens layer 101, a color filter layer 102, a passivation layer 103,a semiconductor layer 106 and a wiring layer 108. At the image-capturingchip 111, the microlens layer 101, the color filter layer 102, thepassivation layer 103, the semiconductor layer 106 and the wiring layer108 are disposed in this order advancing along the Z axis+direction.

The microlens layer 101 includes a plurality of microlenses L. Via eachmicrolens L, light having entered therein is condensed onto aphotoelectric conversion unit 104 to be described later. The colorfilter layer 102 includes a plurality of color filters F. The colorfilters F in the color filter layer 102 are a plurality of differenttypes of color filters with varying spectral characteristics. In morespecific terms, first filters (R) having spectral characteristicswhereby red-color component light is primarily transmitted through,second filters (Gb, Gr) having spectral characteristics wherebygreen-color component light is primarily transmitted through and thirdfilters (B) having spectral characteristics whereby blue-color componentlight is primarily transmitted through are formed in the color filterlayer 102. In the color filter layer 102, the first filters, the secondfilters and the third filters are disposed in, for instance, a Bayerarray. The passivation layer 103, constituted with a nitride film or anoxide film, protects the semiconductor layer 106.

The semiconductor layer 106 includes photoelectric conversion units 104and read-out circuits 105. The semiconductor layer 106 includes aplurality of photoelectric conversion units 104 present between a firstsurface 106 a, which is a light-entry surface and a second surface 106 blocated on the opposite side from the first surface 106 a. The pluralityof photoelectric conversion units 104 are arrayed along the X axis andalong the Y axis in the semiconductor layer 106. The photoelectricconversion units 104 have a photoelectric conversion function throughwhich light is converted to an electric charge. In addition, electriccharges, generated based upon photoelectric conversion signals, areaccumulated in the photoelectric conversion units 104. The photoelectricconversion units 104 may be, for instance, photodiodes. Thesemiconductor layer 106 includes read-out circuits 105 located furthertoward the second surface 106 b relative to the photoelectric conversionunits 104. A plurality of read-out circuits 105 are arrayed along the Xaxis and Y axis in the semiconductor layer 106. The read-out circuits105, each constituted with a plurality of transistors, read out imagedata obtained with electric charges resulting from the photoelectricconversion at the photoelectric conversion units 104 and outputs theimage data thus read out to the wiring layer 108.

The wiring layer 108 includes a plurality of metal layers. The metallayers may be, for instance, Al wirings, Cu wirings and the like. Theimage data read out via the read-out circuits 105 are output to thewiring layer 108. The image data are then output to the signalprocessing chip 112 from the wiring layer 108 via the connectingportions 109.

It is to be noted that the connecting portions 109 may each be providedin correspondence to one of the photoelectric conversion units 104. Asan alternative, the connecting portions 109 may each be provided incorrespondence to a plurality of photoelectric conversion units 104. Thepitch with which the connecting portions 109 are disposed incorrespondence to a plurality of photoelectric conversion units 104 maybe greater than the pitch with which the photoelectric conversion units104 are disposed. In addition, the connecting portions 109 may bedisposed in an area around the area where the photoelectric conversionunits 104 are disposed.

The signal processing chip 112 includes a plurality of signal processingcircuits. The signal processing circuits are engaged in signalprocessing executed on image data output from the image-capturing chip111. Such a signal processing circuit may be, for instance, an amplifiercircuit that amplifies the signal value of an image data signal, acorrelated double sampling circuit engaged in noise reduction processingexecuted to reduce noise in the image data and an analog/digital (A/D)conversion circuit via which analog signals are converted to digitalsignals. The signal processing circuits may each be provided incorrespondence to one of the photoelectric conversion units 104.

As an alternative, a signal processing circuit may be provided incorrespondence to a plurality of photoelectric conversion units 104. Thesignal processing chip 112 includes a plurality of through electrodes(through vias) 110. The through electrodes (through vias) 110 may be,for instance, silicone through electrodes (through-silicon vias). Thecircuits disposed at the signal processing chip 112 are connected withone another via the through electrodes (through vias) 110. The throughelectrodes (through vias) 110 may also be disposed in a peripheral areaat the image-capturing chip 111 and also at the memory chip 113. It isto be noted that some of the elements configuring a signal processingcircuit may be disposed at the image-capturing chip 111. For instance, acomparator that compares an input voltage with a reference voltage in ananalog/digital conversion circuit may be disposed at the image-capturingchip 111 with circuits such as a counter circuit and a latch circuit inthe analog/digital conversion circuit disposed at the signal processingchip 112.

The memory chip 113 includes a plurality of storage units. Image datahaving undergone the signal processing at the signal processing chip 112are stored in the storage units. The storage units may be, for instance,volatile memory circuits such as DRAMs. The storage units may each beprovided in correspondence to one of the photoelectric conversion units104. Alternatively, the storage units may each be provided incorrespondence to a plurality of photoelectric conversion units 104. Theimage data stored in the storage units are output to an image processingunit located at a subsequent stage.

FIG. 3 illustrates the pixel array arrangement and unit areas 131 at theimage-capturing chip 111. More specifically, it shows theimage-capturing chip 111 viewed from the back-side (image-capturingsurface side). At least 20 million pixels, for instance, are arrayed ina matrix pattern in a pixel area. In the example presented in FIG. 3,each unit area 131 is formed with 2 pixels×2 pixels=4 pixels disposedadjacent to each other. The grid lines in the figure indicate a conceptbased upon which adjacent pixels are grouped together to form a unitarea 131. The number of pixels to form a unit area 131 is not limited tothis example and a unit area 131 may instead be formed withapproximately 1000 pixels, e.g., 32 pixels×32 pixels, with more or fewerthan 32×32 pixels, or even with a single pixel.

As the partial enlargement of the pixel area indicates, the unit area131 in FIG. 3 includes four pixels, i.e., green pixels Gb and Gr, a bluepixel B and a red pixel R, disposed therein in an arrangement patterncommonly referred to as a Bayer array pattern. The green pixels Gb andGr each include a green color filter F, and thus receive light in thegreen color wavelength range in the incident light. Likewise, theblue-color pixel B includes a blue color filter F and receives light inthe blue color wavelength range, whereas the red-color pixel R includesa red color filter F and receives light in the red color wavelengthrange.

In the embodiment, a plurality of blocks are defined so that each blockcontains at least one unit area 131. Namely, the minimum unit forconstituting a block is a single unit area 131. As explained earlier,the smallest number of pixels to form a unit area 131, among the valuesthat may be taken for the number of pixels, is one. Accordingly, if ablock is to be defined in units of pixels, the value representing thesmallest number of pixels, among values that may be taken for the numberof pixels to define a single block, will be one. The pixels included inthe individual blocks can be controlled by using control parametervalues different from one another. All the unit areas 131 in a givenblock, i.e., all the pixels present within the particular block, arecontrolled under uniform image-capturing conditions. In other words,photoelectric conversion signals generated under differentimage-capturing conditions can be obtained from a pixel group in a givenblock and from a pixel group included in another block. Examples of suchcontrol parameters include the frame rate, the gain, the culling(subsampling) rate, the number of addition-undergoing rows or columnsover which photoelectric conversion signals are to be added together,the duration or the number of times over which electric charges are tobe accumulated and the number of bits (word length) for digitization.The image sensor 100 allows data to be culled along the direction inwhich columns extend (along the Y axis at the image-capturing chip 111)as well as along the direction in which the rows extend (along the Xaxis at the image-capturing chip 111). In addition, image processingparameters may be used as control parameters.

FIG. 4 illustrates circuits present in a unit area 131. In the examplepresented in FIG. 4, a single unit area 131 is formed with 2 pixels×2pixels=4 pixels disposed adjacent to one another. It is to be noted thatthe number of pixels included in the unit area 131 is not limited tothis example, as explained earlier. Namely, the unit area 131 mayinclude 1000 pixels or more, or it may be made up with a single pixel.Reference signs A through D indicate two-dimensional positions takenwithin the unit area 131.

Pixel reset transistors (RST) present in the unit area 131 adopt astructure that allows them to be turned on/off independently of oneanother in correspondence to the individual pixels. FIG. 4 shows a resetwiring 300, via which the reset transistor for a pixel A is turnedon/off, and a reset wiring 310, via which the reset transistor for pixelB is turn on/off, provided as a member separate from the reset wiring300. Likewise, a reset wiring 320, via which the reset transistor for apixel C is turned on/off, is provided as a member separate from thereset wirings 300 and 310. An exclusive reset wiring 330, via which thereset transistor is turned on/off is also provided for the remainingpixel D.

Pixel transfer transistors (TX) present in the unit area 131 also adopta structure that allows them to be turned on/off independently of oneanother in correspondence to individual pixels. FIG. 4 shows a transferwiring 302 via which the transfer transistor for the pixel A is turnedon/off, a transfer wiring 312 via which the transfer transistor for thepixel B is turned on/off, and a transfer wiring 322 via which thetransfer transistor for the pixel C is turned on/off, provided asmembers independent of one another. An exclusive transfer wiring 332,via which the transfer transistor is turned on/off is also provided forthe remaining pixel D.

In addition, pixel selection transistors (SEL) present in the unit area131 adopt a structure that allow them to be turned on/off theindependently of one another in correspondence to individual pixels.FIG. 4 shows a selection wiring 306 via which the selection transistorfor the pixel A is turned on/off, a selection wiring 316 via which theselection transistor for the pixel B is turned on/off, and a selectionwiring 326 via which the selection transistor for the pixel C is turnedon/off, provided as members independent of one another. An exclusiveselection wiring 336, via which the selection transistor is turnedon/off is also provided for the remaining pixel D.

It is to be noted that a common power wiring 304 is connected to all thepixels A through D included in the unit area 131. Likewise, a commonoutput wiring 308 is connected to all the pixels A through D included inthe unit area 131. In addition, while the power wiring 304 is a commonwiring connected with a plurality of unit areas, output wirings 308 areinstalled each in correspondence to a specific unit area 131. A loadcurrent source 309 supplies an electric current to the output wiring308. The load current source 309 may be disposed on the side where theimage-capturing chip 111 is located or on the side where the signalprocessing chip 112 is located.

By individually turning on/off the reset transistors and the transfertransistors in the unit area 131, the electric charge accumulationparameters including the electric charge accumulation start time, theaccumulation end time and the transfer timing can be controlled for eachof the pixels A through D included in the unit area 131. In addition, byindividually turning on/off the selection transistors in the unit area131 independently of one another, the photoelectric conversion signalsfrom the individual pixels A through D can be output via the commonoutput wiring 308.

The electric charge accumulation at the pixels A through D in the unitarea 131 may be controlled in an order regulated in correspondence torows and columns through a method known as the rolling shutter method inthe related art. In the rolling shutter method, as columns are specifiedafter pixels are selected in correspondence to a specific row,photoelectric conversion signals are output in the order “ABCD” in theexample presented in FIG. 4.

By configuring the circuits in correspondence to unit areas 131 asdescribed above, it is ensured that the charge accumulation time can becontrolled in correspondence to each unit area 131. Namely, theconfiguration makes it possible to individually output photoelectricconversion signals at varying frame rates from one unit area 131 toanother. In addition, by allowing unit areas 131 in another block toidle while electric charge accumulation (image-capturing) is underway inunit areas 131 in a block at the image-capturing chip 111, it ispossible to engage only a specific block at the image-capturing chip 111in image-capturing operation so as to obtain the photoelectricconversion signals output from the particular block. Furthermore,different blocks (accumulation control target blocks) may be selectedfor electric charge accumulation (image-capturing) from one frame toanother, so as to output photoelectric conversion signals from thedifferent blocks successively engaged in image-capturing operation atthe image-capturing chip 111.

As explained earlier, the output wirings 308 are installed each incorrespondence to one of the unit areas 131. Since the image sensor 100is configured by stacking the image-capturing chip 111, the signalprocessing chip 112 and the memory chip 113 one on another, wirings canbe installed without having to increase the size of the individual chipsalong the planar direction by using the electrical connections betweenchips, achieved via the connecting portions 109 for the output wirings308.

<Block Control at the Image Sensor>

The structure achieved in the embodiment makes it possible to setdifferent image capturing conditions for different blocks among theplurality of blocks at the image sensor 32 a. The control unit 34(image-capturing control unit 34 c) engages the image sensor inimage-capturing operation under image-capturing conditions set incorrespondence to each area among the plurality of areas each made tocorrespond to one of the blocks.

FIG. 5 schematically illustrates an image of a subject formed at theimage sensor 32 a in the camera 1. Before an image-capturing instructionis issued, the camera 1 obtains a live view image through photoelectricconversion of the subject image. The term “live view image” is used torefer to a monitor image repeatedly captured at a predetermined framerate (e.g., 60 fps).

Before the imaging field is divided into separate areas by the settingunit 34 b, the control unit 34 sets uniform image-capturing conditionsfor the entire range (i.e., the entire imaging field) of theimage-capturing chip 111. When uniform image-capturing conditions areselected, common image-capturing conditions are set for the entireimaging field. However, even if there is variance in, for instance, theapex value of less than approximately 0.3 (stop), the image-capturingconditions are regarded to be uniform. The image-capturing conditions tobe set uniformly for the entire range of the image-capturing chip 111are determined based upon exposure conditions corresponding to themeasured value representing the subject luminance or exposure conditionsmanually set by the user.

FIG. 5 shows an image that includes a person 61 a, an automobile 62 a, apurse 63 a, mountains 64 a and clouds 65 a, and 66 a, formed at theimage-capturing surface of the image-capturing chip 111. The person 61 ais holding the purse 63 a with both hands. The automobile 62 a is parkedbehind and to the right of the person 61 a.

<Areal Division>

The control unit 34 divides the image plane of the live view image intoa plurality of areas as described below based upon the live view image.First, the object detection unit 34 a detects subject elements in thelive view image. Subject element detection is executed by adopting asubject recognition technology of the known art. The object detectionunit 34 a detects the person 61 a, the automobile 62 a, the purse 63 a,the mountains 64 a, the cloud 65 a, and the cloud 66 a as subjectelements in the example presented in FIG. 5.

Next, the setting unit 34 b divides the live view image plane into areasthat contain the subject elements listed above. The embodiment will bedescribed by designating an area containing the person 61 a as an area61, an area containing the automobile 62 a as an area 62, an areacontaining the person 63 a as an area 63, an area containing themountains 64 a as an area 64, an area containing the cloud 65 a as anarea 65 and an area containing the cloud 66 a as an area 66.

<Setting Image Capturing Conditions for Each Block>

Once the image plane is divided into the plurality of areas via thesetting unit 34 b, the control unit 34 brings up on display at thedisplay unit 35 a setting screen such as that shown in FIG. 6. FIG. 6shows a live view image 60 a brought up on display, with animage-capturing condition setting screen 70 displayed to the rightrelative to the live view image 60 a.

The setting screen 70 includes the frame rate, the shutter speed (TV)and the gain (ISO), set in this order starting from the top,representing examples of image-capturing condition settings. The term“frame rate” is used to refer to the number of live view image framesobtained per second or the number of video image frames recorded by thecamera 1 per second. The gain indicates the ISO sensitivity.Image-capturing conditions other than those in FIG. 6 may be added asneeded as image-capturing condition settings. If all the settings cannotbe fitted within the screen 70, excess settings may be displayed as thesetting field is scrolled up and down.

In the embodiment, the control unit 34 designates an area selected bythe user among the different areas defined by the setting unit 34 b, asan image-capturing condition setting (adjustment) target. For instance,the camera 1 may allow touch-screen operations and in such a case, theuser taps the display position corresponding to a primary photographicsubject for which he wishes to set (adjust) image-capturing conditionson the display screen at the display unit 35 with the live view image 60a currently on display. If the user taps, for instance, the displayposition at which the person 61 a is displayed, the control unit 34designates the area 61 containing the person 61 a in the live view image60 a as an image-capturing condition setting (adjustment) target areaand displays the area 61 by highlighting the edges of the area 61.

The area 61 displayed with highlighted edges (bold-line edges,brightened edges, colored edges, dotted line edges, blinking edges orthe like) in FIG. 6 is the image-capturing condition setting(adjustment) target area. In the example presented in FIG. 6 the liveview image 60 a on display includes the area 61 with highlighted edges.In this situation, the area 61 has been designated as theimage-capturing condition setting (adjustment) target. For instance, theuser of the camera 1 that allows touch-screen operations may tap ashutter speed (TV) display 71, and in response, the control unit 34 willdisplay the current shutter speed setting for the highlighted area(i.e., the area 61) (see reference numeral 68).

While the following explanation is provided in relation to the camera 1by assuming that it allows touch-screen operations, the presentinvention may be adopted in a camera in which image-capturing conditionsare set (adjusted) by operating a button or the like configuring theoperation member 36.

As the user taps either an UP icon 71 a or a DOWN icon 71 b for theshutter speed (TV), the setting unit 34 b raises or lowers the shutterspeed relative to the current setting in the shutter speed display 68 incorrespondence to the tap operation, and also transmits an instructionfor the image-capturing unit 32 (see FIG. 1) so as to adjust theimage-capturing condition for the unit areas 131 (see FIG. 3) at theimage sensor 32 a corresponding to the highlighted area (the area 61),as indicated by the tap operation. An OK icon 72 is an operational iconvia which a selected image-capturing condition is confirmed. The settingunit 34 b sets (adjusts) the frame rate and the gain (ISO) in much thesame way as it does the shutter speed (TV).

It is to be noted that while an explanation has been given on an examplein which the setting unit 34 b sets an image-capturing condition basedupon a user operation, the present invention is not limited to thisexample. Instead of setting the image-capturing conditions in responseto a user operation, the setting unit 34 b may set an image-capturingcondition based upon a decision made by the control unit 34. Forinstance, white-clipping (blown-out highlights) or black-clipping(crushed blacks) may have occurred in an area that includes a subjectelement achieving the highest or lowest luminance in the image and insuch a case, the setting unit 34 b may select an image-capturingcondition so as to eliminate the white-clipping or black-clipping basedupon a decision made by the control unit 34.

The initial image-capturing condition settings are sustained for areasthat are not highlighted (areas other than the area 61).

Instead of highlighting the edges of the area designated as theimage-capturing condition setting (adjustment) target, the control unit34 may brighten the entire target area, may display the target area byincreasing the contrast for the entire area or may flash the entiretarget area on display. As an alternative, it may frame the target area.The frame around the target area may be a double frame or a singleframe, and the display mode for the frame around the target area, suchas the type of line representing the frame, the color, the brightness orthe like may be adjusted as needed. In addition, the control unit 34 maybring up on display a symbol such as an arrow pointing to theimage-capturing condition setting target area in the vicinity of thetarget area. The control unit 34 may darken the areas other than theimage-capturing condition setting (adjustment) target area or maydisplay the areas other than the target area by lowering the contrast.

Once the image-capturing conditions for the individual areas have beenset as described above, the shutter release button (not shown)constituting the operation member 36 or an image-capturing startinstruction icon (shutter release icon) on display is operated, thecontrol unit 34 controls the image-capturing unit 32 in response, so asto engage it in image-capturing operation under the image-capturingconditions individually set for the different areas. The imageprocessing unit 33 then executes the image processing on image dataobtained via the image-capturing unit 32. As explained earlier, theimage processing can be executed under different image processingconditions set for different areas.

Once the image processing has been executed by the image processing unit33, the recording unit 37, having received an instruction from thecontrol unit 34, records the image data having undergone the imageprocessing into the recording medium constituted with, for instance, amemory card (not shown). A sequence of image-capturing processing thusends.

<Data Selection Processing>

As described above, once the setting unit 34 b has divided the imagingfield into a plurality of areas, image-capturing conditions can be set(adjusted) for an area selected by the user or for an area determined bythe control unit 34. If different image-capturing conditions have beenset for individual areas, the control unit 34 initiates the followingdata selection processing as needed.

1. When Executing Image Processing

If a specific type of image processing is to be executed on image dataobtained by applying different image-capturing conditions for differentareas, the image processing unit 33 (selection unit 33 b) executes dataselection processing as preliminary processing preceding the imageprocessing on image data located near a boundary between areas. Throughthe specific type of image processing, image data are generated throughcalculation in correspondence to a processing target position set in theimage by referencing image data available at a plurality of referencepositions around the target position, and in more specific terms, such aspecific type of image processing may be, for instance, defective pixelcorrection processing, color interpolation processing, edge enhancementprocessing or noise reduction processing.

The data selection processing is executed so as to lessen any unnaturalappearance that is bound to occur in the processed image due to thedifference in the image-capturing conditions set for divided areas. Thetarget position may be located near the boundary between divided areas,and under such circumstances, image data obtained by applyingimage-capturing conditions matching those corresponding to image data atthe target position and image data obtained by applying image-capturingconditions different from those corresponding to image data at thetarget position may both be present at the plurality of referencepositions around the target position. In the embodiment, based upon thetheory that it is more desirable to generate image data at the targetposition through calculation executed by referencing the image data atreference positions at which image-capturing conditions matching thosefor the target position have been applied rather than by referencing theimage data at reference positions at which different image-capturingconditions have been applied, data to be used in the image processingare selected as described below.

FIG. 7(a) shows an area 80 near the boundary between the area 61 and thearea 64 in the live view image 60 a. In this example, firstimage-capturing conditions have been set at least for the area 61 thatincludes the person and second image-capturing conditions have been setfor the area 64 that includes the mountains. FIG. 7(b) provides anenlargement of the area 80 near the boundary in FIG. 7(a). Image datafrom the pixels on the image sensor 32 a, corresponding to the area 61for which the first image-capturing conditions have been set, areindicated as unshaded squares, whereas image data from the pixels on theimage sensor 32 a, corresponding to the area 64 for which the secondimage-capturing conditions have been set, are indicated as shadedsquares. In FIG. 7(b), image data from a target pixel P are located inthe area 61, at a boundary area near a boundary 81 between the area 61and the area 64. Pixels around the target pixel P, present within apredetermined range 90 centered on the target pixel P (e.g., a 3×3 pixelrange) are designated as reference pixels (eight pixels in thisexample). FIG. 7(c) provides an enlarged view of the target pixel P andthe reference pixels. The position taken by the target pixel P is thetarget position, whereas the positions taken by the reference pixelssurrounding the target pixel P are reference positions.

The image processing unit 33 (generation unit 33 c) may execute imageprocessing by directly referencing the image data at the referencepixels without first executing the data selection processing. However,if the image-capturing conditions (first image-capturing conditions)applied at the target pixel P are different from the image-capturingconditions (second image-capturing conditions) applied at referencepixels around the target pixel P, the selection unit 33 b selects imagedata obtained under the first image-capturing conditions, among the setsof image data at the reference pixels, as the image data to be used inthe image processing, as in (example 1) through (example 3) describedbelow. The generation unit 33 c then executes the image processing so asto generate image data for the target pixel P through calculation byreferencing the image data at the reference pixels selected through thedata selection processing. In FIG. 7(c), the data output from theunshaded pixels are image data obtained under the first image-capturingconditions, whereas the data output from the shaded pixels are imagedata obtained under the second image-capturing conditions. In theembodiment, the image data obtained under the second image-capturingconditions are not selected and thus, the image data output from theshaded pixels are not used in the image processing.

Example 1

If, for instance, the first image-capturing conditions and the secondimage-capturing conditions are different from each other only in the ISOsensitivity with an ISO sensitivity setting of 100 selected for thefirst image-capturing conditions and an ISO sensitivity setting of 800selected for the second image-capturing conditions, the image processingunit 33 (selection unit 33 b) selects image data obtained under thefirst image-capturing conditions, among the sets of image data at thereference pixels, as the image data to be used in the image processing.In other words, the image data obtained under the second image-capturingconditions different from the first image-capturing conditions, amongthe sets of image data at the reference pixels, are not used in theimage processing.

Example 2

If the first image-capturing conditions and the second image-capturingconditions are different from each other only in the shutter speed witha shutter speed setting of 1/1000 sec selected for the firstimage-capturing conditions and a shutter speed setting of 1/100 secselected for the second image-capturing conditions, the image processingunit 33 (selection unit 33 b) selects image data obtained under thefirst image-capturing conditions, among the sets of image data at thereference pixels, as the image data to be used in the image processing.In other words, the image data obtained under the second image-capturingconditions different from the first image-capturing conditions, amongthe sets of image data at the reference pixels, are not used in theimage processing.

Example 3

If the first image-capturing conditions and the second image-capturingconditions are different from each other only in the frame rate (withuniform electric charge accumulation time) with a frame rate setting of30 fps selected for the first image-capturing conditions and a framerate setting of 60 fps selected for the second image-capturingconditions, the image processing unit 33 (selection unit 33 b) selectsimage data obtained under the second image-capturing conditions (60fps), among the sets of image data at the reference pixels, whichexpress frame images obtained with timing close to the timing with whichframe images have been obtained under the first image-capturingconditions (30 fps). Namely, image data expressing frame images obtainedwith timing significantly different from the timing with which the frameimages have been obtained under the first image-capturing conditions (30fps) among the sets of image data at the reference pixels, are not usedin the image processing.

If, on the other hand, the image-capturing conditions applied at thetarget pixel P and the image-capturing conditions applied at all thereference pixels around the target pixel P match completely, the imageprocessing unit 33 (selection unit 33 b) selects all the image data. Forinstance, upon deciding that image-capturing conditions identical to theimage-capturing conditions set for the target pixel P have been appliedat all the reference pixels, the image processing unit executes theimage processing so as to generate image data for the target pixel Pthrough calculation by directly referencing the image data at all thereference pixels.

It is to be noted that as has been explained earlier, image-capturingconditions indicating a slight difference (e.g., apex values differentfrom each other by 0.3 (stop) or less) may be regarded as uniformimage-capturing conditions.

<Examples of Image Processing>

Examples of image processing that may be executed in conjunction withdata selection processing will be explained next.

(1) Defective Pixel Correction Processing

Defective pixel correction processing is a type of image processingexecuted during image-capturing operation in the embodiment. Generally,a pixel defect may occur during or following the production processthrough which the image sensor 100, which is a solid-state image sensor,is manufactured, and image data may be output from a defective pixel atan abnormal level. Accordingly, the image processing unit 33 (generationunit 33 c) corrects image data output from a defective pixel so as torender the image data at the pixel position at which a pixel defect hasoccurred, less conspicuous.

An example of defective pixel correction processing will be describedbelow. The image processing unit 33 (generation unit 33 c) designates,for instance, a pixel in an image corresponding to a single frame, whichtakes a defective pixel position recorded in advance in a nonvolatilememory (not shown), as a target pixel P (processing target pixel) andalso designates pixels (eight pixels in this example) present around thetarget pixel P within a predetermined range 90 (e.g., a 3×3 pixel range)centered around the target pixel P (see FIG. 7(c)) as reference pixels.

The image processing unit 33 (generation unit 33 c) calculates a largestvalue and a smallest value among the values indicated in the image dataobtained at the reference pixels, and if the image data output from thetarget pixel P indicate a value greater than the largest value orsmaller than the smallest value, it executes Max, Min filter processingso as to replace the image data output from the target pixel P eitherwith the image data indicating the largest value or the smallest value.This processing is executed for all the defective pixels, the positioninformation for which is recorded in the nonvolatile memory.

The image processing unit 33 (selection unit 33 b) in the embodimentselects image data, among the sets of image data at the referencepixels, obtained under the first image-capturing conditions, if thereference pixels include pixels to which the second image-capturingconditions, different from the first image-capturing conditions appliedat the target pixel P, have been applied. Then the image processing unit33 (generation unit 33 c) executes the Max, Min filter processing, asdescribed above, by referencing the selected image data. It is to benoted that the defective pixel correction processing may be executed byaveraging the values in the selected image data.

(2) Color Interpolation Processing

Color interpolation processing is a type of image processing executedduring image-capturing operation in the embodiment. As shown in FIG. 3,green-color pixels Gb and Gr, blue-color pixels B and red-color pixels Rare disposed in a Bayer array at the image-capturing chip 111 in theimage sensor 100. Since image data with color components different fromthe color component corresponding to the color filter disposed at agiven pixel position are missing at the particular pixel position, theimage processing unit 33 (generation unit 33 c) executes colorinterpolation processing so as to generate image data for the missingcolor components by referencing image data at positions taken bysurrounding pixels.

An example of color interpolation processing will be explained. FIG.8(a) shows how image data output from the image sensor 100 may bearranged. In line with the Bayer array rules, sets of image datacorresponding to the various pixel positions each take a specific coloramong R, G and B.

<G Color Interpolation>

Standard G color interpolation will be described first. The imageprocessing unit 33 (generation unit 33 c) engaged in G colorinterpolation sequentially designates positions each corresponding tothe R color component or the B color component as a target position andgenerates G color component image data for the target position byreferencing four sets of G color component image data at referencepositions around the target position. For instance, when generating Gcolor component image data for the target position framed with boldlines in FIG. 8(b), (i.e., a second-row/second-column position, countingfrom the upper left corner. In the following description, the targetposition will be likewise indicated in relation to the row and thecolumn it occupies relative to the upper left corner position), the foursets of G color component image data G1 through G4 located near thetarget position (second row/second column) are referenced. The imageprocessing unit 33 (generation unit 33 c) may generate, for instance, Gcolor component image data (aG1+bG2+cG3+dG4)/4 for the target position(second row/second column). It is to be noted that a through d areweighting coefficients each selected in correspondence to the distancebetween the specific reference position and the target position and theimage structure.

Next, the G color interpolation executed in the embodiment will beexplained. In FIG. 8(a) through FIG. 8(c) the first image-capturingconditions are set for the area located to the left of and above thebold lines and the second image-capturing conditions are set for thearea located to the right of and below the bold lines. It is to be notedthat the first image-capturing conditions and the second image-capturingconditions set as shown in FIG. 8(a) through FIG. 8(c) are differentfrom each other. In addition, the G color component image data G1through G4 in FIG. 8(b) are located at positions designated as referencepositions referenced in the image processing executed for the pixel atthe target position (second row/second column). The firstimage-capturing conditions are set for the target position (secondrow/second column) in FIG. 8(b). The image data G1 through G3 located atsome of the reference positions have been obtained under the firstimage-capturing conditions. However, the image data G4 located at one ofthe reference positions have been obtained under the secondimage-capturing conditions. Accordingly, the image processing unit 33(selection unit 33 b) selects the image data G1 through G3 obtainedunder the first image-capturing conditions among the G color componentimage data G1 through G4. The image processing unit 33 (generation unit33 c) executes calculation so as to generate G color component imagedata for the target position (second row/second column) by referencingthe image data selected as described above. The image processing unit 33(generation unit 33 c) may generate, for instance, G color componentimage data (a1G1+b1G2+c1G3)/3 for the target position (second row/secondcolumn). It is to be noted that a1 through c1 are weighting coefficientseach selected in correspondence to the distance between the specificreference position and the target position and the image structure.

The image processing unit 33 (generation unit 33 c) is able to obtain Gcolor component image data for each pixel position, as shown in FIG.8(c) by generating G color component image data at each position amongthe B color component positions and the R color component positions inFIG. 8(a).

<R Color Interpolation>

FIG. 9(a) shows only the R color component image data extracted from theimage data in the diagram in FIG. 8(a). The image processing unit 33(generation unit 33 c) generates, through calculation, image datarepresenting a chrominance component Cr in FIG. 9(b), based upon the Gcolor component image data shown in FIG. 8(c) and the R color componentimage data shown in FIG. 9(a).

Standard chrominance component Cr interpolation will be described first.For instance, when generating chrominance component image data Crcorresponding to the chrominance component Cr for the target position(second-row/second-column) indicated for the target position framed withbold lines in FIG. 9(b), four sets of chrominance component image dataCr1 through Cr4 located near the target position (second row/secondcolumn) are referenced. The image processing unit 33 (generation unit 33c) may generate, for instance, chrominance component image data Cr(eCr1+fCr2+gCr3+hCr4)/4 for the target position (second row/secondcolumn). It is to be noted that e through h are weighting coefficientseach selected in correspondence to the distance between the specificreference position and the target position and the image structure.

Likewise, when generating image data corresponding to the chrominancecomponent Cr for the target position (second-row/third-column) framedwith bold lines in FIG. 9(c), four sets of chrominance component imagedata Cr2 and Cr4 through Cr6 located near the target position (secondrow/third column) are referenced. The image processing unit 33(generation unit 33 c) may generate, for instance, chrominance componentimage data Cr (qCr2+rCr4+sCr5+tCr6)/4 for the target position (secondrow/third column). It is to be noted that q through t are weightingcoefficients each selected in correspondence to the distance between thespecific reference position and the target position and the imagestructure. Chrominance component image data Cr are generated for eachpixel position through the process described above.

Next, the chrominance component Cr interpolation executed in theembodiment will be explained. In FIG. 9(a) through FIG. 9(c) the firstimage-capturing conditions are set for the area located to the left ofand above the bold lines and the second image-capturing conditions areset for the area located to the right of and below the bold lines. It isto be noted that the first image-capturing conditions and the secondimage-capturing conditions set as shown in FIG. 9(a) through FIG. 9(c)are different from each other. The position indicated with the bold-lineframe (second row/second column) in FIG. 9(b) is the target position forthe chrominance component Cr interpolation. In addition, the chrominancecomponent image data Cr1 through Cr4 in FIG. 9(b) are located atpositions designated as reference positions in the image processingexecuted for the pixel at the target position (second row/secondcolumn). The first image-capturing conditions are set for the targetposition (second row/second column) in FIG. 9(b). The image data Cr1,Cr3 and Cr4 located at some of the reference positions have beenobtained under the first image-capturing conditions. However, the imagedata Cr2 located at one of the reference positions have been obtainedunder the second image-capturing conditions. Accordingly, the imageprocessing unit 33 (selection unit 33 b) selects the image data Cr1, Cr3and Cr4 obtained under the first image-capturing conditions among thechrominance component image data Cr1 through Cr4. The image processingunit 33 (generation unit 33 c) then executes calculation so as togenerate chrominance component image data Cr for the target position(second row/second column) by referencing the image data selected asdescribed above. The image processing unit 33 (generation unit 33 c) maygenerate, for instance, chrominance component image data Cr(e1Cr1+g1Cr3+h1Cr4)/3 for the target position (second row/secondcolumn). It is to be noted that e1, g1 and h1 are weighting coefficientseach selected in correspondence to the distance between the specificreference position and the target position and the image structure.

In addition, the target position for the chrominance component Crinterpolation is the position indicated with the bold-line frame (secondrow/third column) in FIG. 9(c). The chrominance component image dataCr2, Cr4, Cr5 and Cr6 in FIG. 9(c) are located at positions designatedas reference positions in the image processing executed for the pixel atthe target position (second row/third column). The secondimage-capturing conditions are set for the target position (secondrow/third column) in FIG. 9(c). The image data Cr4 and Cr5 located atsome of the reference positions have been obtained under the firstimage-capturing conditions. However, the image data Cr2 and Cr6 locatedat the other reference positions have been obtained under the secondimage-capturing conditions.

Accordingly, the image processing unit 33 (selection unit 33 b) selectsthe image data Cr2 and Cr6 obtained under the second image-capturingconditions among the chrominance component image data Cr2 and Cr4through Cr6 based upon a principle similar to that described inreference to FIG. 9(b). The image processing unit 33 (generation unit 33c) then executes calculation so as to generate chrominance componentimage data Cr for the target position (second row/third column) byreferencing the image data selected as described above. The imageprocessing unit 33 (generation unit 33 c) may generate, for instance,chrominance component image data (g2Cr2+h2Cr6)/2 for the target position(second row/third column). It is to be noted that g2 and h2 areweighting coefficients each selected in correspondence to the distancebetween the specific reference position and the target position and theimage structure.

Once chrominance component image data Cr have been obtained for eachpixel position, the image processing unit 33 (generation unit 33 c) isable to obtain R color component image data corresponding to each pixelposition by adding each set of G color component image data in FIG. 8(c)to the chrominance component image data Cr at the corresponding pixelposition.

<B Color Interpolation>

FIG. 10(a) shows only the B color component image data extracted fromthe image data in the diagram in FIG. 8(a). The image processing unit 33(generation unit 33 c) generates, through calculation, image datarepresenting a chrominance component Cb in FIG. 10(b), based upon the Gcolor component image data shown in FIG. 8(c) and the B color componentimage data shown in FIG. 10(a).

Standard chrominance component Cb interpolation will be described first.For instance, when generating image data corresponding to thechrominance component image data Cb for the target position(third-row/third-column) framed with bold lines in FIG. 10(b), four setsof chrominance component image data Cb1 through Cb4 located near thetarget position (third row/third column) are referenced. The imageprocessing unit 33 (generation unit 33 c) may generate, for instance,chrominance component image data Cb (uCb1+vCb2+wCb3+xCb4)/4 for thetarget position (third row/third column). It is to be noted that uthrough x are weighting coefficients each selected in correspondence tothe distance between the specific reference position and the targetposition and the image structure.

Likewise, when generating chrominance component image data Cb for thetarget position (third-row/fourth-column) framed with bold lines in FIG.10(c), four sets of chrominance component image data Cb2 and Cb4 throughCb6 located near the target position (third row/fourth column) arereferenced. The image processing unit 33 (generation unit 33 c) maygenerate, for instance, chrominance component image data Cb(yCb2+zCb4+αCb5+βCb6)/4 for the target position (third row/fourthcolumn). It is to be noted that y, z, α and β are weighting coefficientseach selected in correspondence to the distance between the specificreference position and the target position and the image structure.Chrominance component image data Cb are generated for each pixelposition through the process described above.

Next, the chrominance component Cb interpolation executed in theembodiment will be explained. In FIG. 10(a) through FIG. 10(c), thefirst image-capturing conditions are set for the area located to theleft of and above the bold lines and the second image-capturingconditions are set for the area located to the right of and below thebold lines. It is to be noted that the first image-capturing conditionsand the second image-capturing conditions set as shown in FIG. 10(a)through FIG. 10(c) are different from each other. The position indicatedwith the bold-line frame (third row/third column) in FIG. 10(b) is thetarget position for the chrominance component Cb interpolation. Inaddition, the chrominance component image data Cb1 through Cb4 in FIG.10(b) are located at positions designated as reference positions in theimage processing executed for the pixel at the target position (thirdrow/third column). The second image-capturing conditions are set for thetarget position (third row/third column) in FIG. 10(b). The image dataCb1 and Cb3 located at some of the reference positions have beenobtained under the first image-capturing conditions. However, the imagedata Cb2 and Cb4 located at the other reference positions have beenobtained under the second image-capturing conditions. Accordingly, theimage processing unit 33 (selection unit 33 b) selects the image dataCb2 and Cb4 obtained under the second image-capturing conditions amongthe chrominance component image data Cb1 through Cb4. The imageprocessing unit 33 (generation unit 33 c) then executes calculation soas to generate chrominance component image data Cb for the targetposition (third row/third column) by referencing the image data selectedas described above. The image processing unit 33 (generation unit 33 c)may generate, for instance, chrominance component image data Cb(v1Cb2+x1Cb4)/2 for the target position (third row/third column). It isto be noted that v1 and x1 are weighting coefficients each selected incorrespondence to the distance between the specific reference positionand the target position and the image structure.

Chrominance component image data Cb are generated in much the same wayfor the target position (third row/fourth column) indicated with thebold-line frame in FIG. 10(c).

The target position for the chrominance component Cb interpolation isthe position indicated with the bold-line frame (third row/fourthcolumn) in FIG. 10(c). In addition, the chrominance component image dataCb2 and Cb4 through Cb6 in FIG. 10(c) are located at positionsdesignated as reference positions in the image processing executed forthe pixel at the target position (third row/fourth column). The secondimage-capturing conditions are set for the target position (thirdrow/fourth column) in FIG. 10(c). Furthermore, the image data Cb2 andCb4 through Cb6 at all the reference positions have been generated underthe second image-capturing conditions. Accordingly, the image processingunit 33 (generation unit 33 c) generates chrominance component imagedata Cb for the target position (third row/fourth column) by referencingthe four sets of chrominance component image data Cb2 and Cb4 throughCb6 located near the target position (third row/fourth column).

Once chrominance component image data Cb have been obtained for eachpixel position, the image processing unit 33 (generation unit 33 c) isable to obtain B color component image data in correspondence to eachpixel position by adding each set of G color component image data inFIG. 8(c) to the chrominance component image data Cb at thecorresponding pixel position.

(3) Edge Enhancement Processing

An example of edge enhancement processing will be explained next. Theimage processing unit 33 (generation unit 33 c) executes, for instance,a linear filter operation of the known art for a single image frame byusing a kernel of a predetermined size centered on a target pixel P(processing target pixel). Assuming that a sharpening filterrepresenting an example of a linear filter takes a kernel size of N×Npixels, the position taken by the target pixel P is the target positionand the positions taken by (N²−1) reference pixels surrounding thetarget pixel P are reference positions.

It is to be noted that the kernel size may be N pixels×M pixels instead.

The image processing unit 33 (generation unit 33 c) executes filterprocessing through which the image data at the target pixel P arereplaced with linear filter operation results starting, for instance, onthe horizontal line at the top of the frame image and moving toward thehorizontal line at the bottom by shifting the target pixel from left toright on each horizontal line.

If the reference pixels described above include pixels to which secondimage-capturing conditions different from first image-capturingconditions applied at the target pixel P are applied, the imageprocessing unit 33 (selection unit 33 b) in the embodiment selects imagedata obtained under the first image-capturing conditions among the setsof image data at the reference pixels. Subsequently, the imageprocessing unit 33 (generation unit 33 c) executes the linear filterprocessing mentioned earlier by referencing the selected image data.

(4) Noise Reduction Processing

An example of noise reduction processing will be explained next. Theimage processing unit 33 (generation unit 33 c) may execute, forinstance, a linear filter operation of the known art for, for instance,a single image frame by using a kernel of a predetermined size centeredon a target pixel P (processing target pixel). Assuming that a smoothingfilter representing an example of a linear filter takes a kernel size ofN×N pixels, the position taken by the target pixel P is the targetposition and the positions taken by (N²−1) reference pixels surroundingthe target pixel P are reference positions.

It is to be noted that the kernel size may be N×M pixels instead.

The image processing unit 33 (generation unit 33 c) executes filterprocessing through which the image data at the target pixel P arereplaced with linear filter operation results starting, for instance, onthe horizontal line at the top of the frame image and moving toward thehorizontal line at the bottom by shifting the target pixel from left toright on each horizontal line.

If the reference pixels described above include pixels to which secondimage-capturing conditions different from first image-capturingconditions applied at the target pixel P are applied, the imageprocessing unit 33 (selection unit 33 b) in the embodiment selects imagedata obtained under the first image-capturing conditions among the setsof image data at the reference pixels. Subsequently, the imageprocessing unit 33 (generation unit 33 c) executes the linear filterprocessing mentioned earlier by referencing the selected image data.

As described above, if the reference pixels include pixels to which thesecond image-capturing conditions different from the firstimage-capturing conditions applied at the target pixel P are applied,the image processing unit 33 (selection unit 33 b) selects image dataobtained under the first image-capturing conditions among the sets ofimage data obtained at the reference pixels. Subsequently, the imageprocessing unit 33 (generation unit 33 c) executes image processing suchas defective pixel correction processing, color interpolationprocessing, edge enhancement processing or noise reduction processing byreferencing the selected image data. The image processing unit 33(selection unit 33 b) generates an image by executing similar processingfor image data (including image data obtained under the secondimage-capturing conditions) originating from other pixels, output fromthe image sensor. The image thus generated is brought up on display at adisplay unit such as a display device.

2. When Executing Focus Detection Processing

The control unit 34 (AF operation unit 34 d) executes focus detectionprocessing by using image data corresponding to a specific position(focus detection position) on the imaging field. If differentimage-capturing conditions have been set for different areas and thefocus detection position for an AF operation is located at a boundarybetween areas, the control unit 34 (AF operation unit 34 d) executesdata selection processing on image data obtained for purposes of focusdetection, which are located near the boundary of the areas, aspreliminary processing prior to the focus detection processing.

The data selection processing is executed in order to minimize theextent to which the accuracy of the focus detection processing islowered due to different image-capturing conditions set for differentareas in the imaging field divided by the setting unit 34 b. Forinstance, the image data to be used for focus detection available at thefocus detection position set to detect an image shift quantityrepresenting the extent of image misalignment (phase difference) in theimage, may be located near a boundary between areas and in such a case,the focus detection image data may include image data obtained underdifferent image-capturing conditions. In the embodiment, data selectionprocessing is executed as described below based upon the theory that itis better to detect an image shift quantity (phase difference) by usingimage data obtained under uniform image-capturing conditions rather thanto detect an image shift quantity (phase difference) by using image dataobtained under different image-capturing conditions.

<Example of Focus Detection Processing>

An example of focus detection processing executed in conjunction withdata selection processing will be described. Through the AF operationexecuted in the embodiment, focus is adjusted on a subject correspondingto, for instance, a focus detection position selected by the user from aplurality of focus detection positions. The control unit 34 (AFoperation unit 34 d) calculates a defocus quantity representing theextent of defocusing in the image-capturing optical system 31 bydetecting the image shift quantity (phase difference) representing theextent of image misalignment among a plurality of subject images formedwith light fluxes having passed through different pupil areas at theimage-capturing optical system 31. The control unit 34 (AF operationunit 34 d) then drives the focus lens in the image-capturing opticalsystem 31 to a position at which the defocus quantities is reduced to 0(equal to or less than an allowable value), i.e., to the in-focusposition.

FIG. 11 presents an example of positional arrangement with which focusdetection pixels may be disposed at the image-capturing surface of theimage sensor 32 a. In the embodiment, focus detection pixels aredisposed in series at discrete positions along the X axis (along thehorizontal direction) at the image-capturing chip 111. In the examplepresented in FIG. 11, 15 focus detection pixel lines 160 are set over apredetermined interval. Focus detection pixels making up focus detectionpixel lines 160 each output a photoelectric conversion signal to be usedfor focus detection. Standard image-capturing pixels are disposed atpixel positions other than those on the focus detection pixel lines 160at the image-capturing chip 111. The image-capturing pixels outputphotoelectric conversion signals used to generate a live view image oran image to be recorded.

FIG. 12 is an enlarged view of part of the focus detection pixel line160 corresponding to a focus detection position 80A in FIG. 11. FIG. 12shows red-color pixels R, green-color pixels G (Gb, Gr) blue-colorpixels B, focus detection pixels S1 and focus detection pixels S2. Thered-color pixels R, the green-color pixels G (Gb, Gr) and the blue-colorpixels B are disposed in conformance to the Bayer array rules describedearlier.

The square areas representing the red-color pixels R, the green-colorpixels G (Gb, Gr) and the blue-color pixels B each indicate a lightreceiving area of an image-capturing pixel. The individualimage-capturing pixels each receive a light flux having passed throughan exit pupil in the image-capturing optical system 31 (see FIG. 1). Inother words, the red-color pixels R, the green-color pixels G (Gb, Gr)and the blue-color pixels B each include a square-shaped mask openingand light having passed through the mask openings reaches the lightreceiving portions at the image-capturing pixels.

It is to be noted that the shape of the light receiving areas (maskopenings) at the red-color pixels R, the green-color pixels G (Gb, Gr)and the blue-color pixels B does not need to be quadrangular and thelight receiving areas may be formed in a round shape.

The semicircular areas in the focus detection pixels S1 and the focusdetection pixels S2 each indicate a light receiving area at a focusdetection pixel. Namely, the focus detection pixels S1 each include asemicircular mask opening located on the left side of the pixel in FIG.12, and light having passed through the mask openings reaches the lightreceiving portions at the focus detection pixels S1. The focus detectionpixels S2, on the other hand, each include a semicircular mask openinglocated on the right side of the pixel in FIG. 12, and light havingpassed through the mask openings reaches the light receiving portions atthe focus detection pixels S2. Thus, a pair of light fluxes passingthrough different areas of the exit pupil in the image-capturing opticalsystem 31 (see FIG. 1) are received, one at the focus detection pixelsS1 and the other at the focus detection pixels S2.

It is to be noted that the positions taken by the focus detection pixellines 160 at the image-capturing chip 111 are not limited to those inthe example presented in FIG. 11. In addition, the number of focusdetection pixel lines 160 is not limited to the example presented inFIG. 11. Furthermore, the mask openings at the focus detection pixels S1and the focus detection pixels S2 may assume, for instance, arectangular shape achieved by splitting the square light receiving area(mask opening) at an image-capturing pixel R, an image-capturing pixel Gor an image-capturing pixel B along the lateral direction, instead ofthe semicircular shape.

Moreover, focus detection pixel lines 160 may each be formed bydisposing focus detection pixels one after another along the Y axis(along the vertical direction) at the image-capturing chip 111, instead.An image sensor configured with image-capturing pixels and focusdetection pixels disposed in a two-dimensional array as shown in FIG. 12is of the known art, and detailed illustrations and a detailedexplanation of these pixels are not provided.

It is to be noted that each focus detection pixel S1 or S2 receives onelight flux in the pair of focus detection light fluxes in the examplepresented in FIG. 12. As an alternative, both light fluxes in the pairof focus detection light fluxes may be received at each focus detectionpixel. By allowing each focus detection pixel to receive both lightfluxes in the pair of focus detection light fluxes, the photoelectricconversion signals obtained from the focus detection pixels can be alsoused as photoelectric conversion signals to be recorded.

The control unit 34 (AF operation unit 34 d) detects the image shiftquantity (phase difference) representing the extent of imagemisalignment occurring between a pair of images formed with the pair oflight fluxes passing through different areas in the image-capturingoptical system 31 (see FIG. 1) based upon focus detection photoelectricconversion signals (signal data) output from the focus detection pixelsS1 and the focus detection pixels S2. Then, it executes an arithmeticoperation to calculate a defocus quantity based upon the image shiftquantity (phase difference). Since the arithmetic operation executed tocalculate the defocus quantity through this split pupil phase method isof the known art in the field of cameras, a detailed explanation is notprovided.

It is assumed that the focus detection position 80A (see FIG. 11),located at a position corresponding to the area 80 near the boundary ofthe area 61 in the live view image 60A such as that shown in FIG. 7(a)has been selected by the user. FIG. 13 presents an enlarged view of thefocus detection position 80A. The first image-capturing conditions havebeen set for the unshaded pixels, whereas the second image-capturingconditions have been set for the shaded pixels. The area enclosed by aframe 170 in FIG. 13 corresponds to the focus detection pixel line 160(see FIG. 11).

The control unit 34 (AF operation unit 34 d) normally executes focusdetection processing by directly using focus detection signal dataoutput from the focus detection pixels within the frame 170 withoutexecuting data selection processing. However, if the focus detectionsignal data from the area enclosed by the frame 170 include both focusdetection signal data obtained under the first image-capturingconditions and focus detection image data obtained under the secondimage-capturing conditions, the control unit 34 (AF operation unit 34 d)selects the focus detection signal data obtained under the firstimage-capturing conditions, among the sets of focus detection signaldata available in the area enclosed by the frame 170, as focus detectionsignal data to be used for the focus detection processing, as explainedin reference to (example 1) through (example 3) below. The control unit34 (AF operation unit 34 d) then executes the focus detection processingby using the focus detection signal data selected through the dataselection processing. In FIG. 13, the data output from the unshadedpixels are focus detection signal data obtained under the firstimage-capturing conditions and the data output from the shaded pixelsare focus detection signal data obtained under the secondimage-capturing conditions. In this embodiment, the focus detectionsignal data obtained under the second image-capturing conditions are notselected and thus the focus detection signal data output from the shadedaxles are not used in the focus detection processing.

Example 1

If, for instance, the first image-capturing conditions and the secondimage-capturing conditions are different from each other only in the ISOsensitivity with an ISO sensitivity setting of 100 selected for thefirst image-capturing conditions and an ISO sensitivity setting of 800selected for the second image-capturing conditions, the control unit 34(AF operation unit 34 d) selects focus detection signal data obtainedunder the first image-capturing conditions, among the sets of image dataavailable in the area enclosed by the frame 170, as the signal data tobe used in the focus detection processing. In other words, the focusdetection signal data obtained under the second image-capturingconditions different from the first image-capturing conditions, amongthe focus detection signal available in the area enclosed by the frame170, are not used in the focus detection processing.

Example 2

If the first image-capturing conditions and the second image-capturingconditions are different from each other only in the shutter speed witha shutter speed setting of 1/1000 sec selected for the firstimage-capturing conditions and a shutter speed setting of 1/100 secselected for the second image-capturing conditions, the control unit 34(AF operation unit 34 d) selects focus detection signal data obtainedunder the first image-capturing conditions, among the sets of focusdetection signal data available in the area enclosed with the frame 170,to be used in the focus detection processing. In other words, the focusdetection signal data obtained under the second image-capturingconditions different from the first image-capturing conditions, amongthe sets of focus detection signal data available in the area enclosedby the frame 170, are not used in the focus detection processing.

Example 3

If the first image-capturing conditions and the second image-capturingconditions are different from each other only in the frame rate (withuniform electric charge accumulation time) with a frame rate setting of30 fps selected for the first image-capturing conditions and a framerate setting of 60 fps selected for the second image-capturingconditions, the control unit 34 (AF operation unit 34 d) selects focusdetection signal data obtained under the first image-capturingconditions among the sets of focus detection signal data available inthe area enclosed by the frame 170. Namely, the focus detection signaldata obtained under the second image-capturing conditions with timingdifferent from the timing with which the image data have been obtainedunder the first image-capturing conditions, among the sets of focusdetection signal data available in the area enclosed by the frame 170,are not used in the focus detection processing.

However, if the focus detection signal data in the area enclosed by theframe 170 have been obtained under uniform image-capturing conditions,the control unit 34 (AF operation unit 34 d) does not execute the dataselection processing described above. Namely, the control unit 34 (AFoperation unit 34 d) executes focus detection processing by directlyutilizing the focus detection signal data output from the focusdetection pixels in the area enclosed by the frame 170.

It is to be noted that as has been explained earlier, image-capturingconditions indicating a slight difference from each other are regardedas uniform image-capturing conditions.

In addition, while focus detection signal data obtained under the firstimage-capturing conditions are selected from the sets of focus detectionsignal data available in the area enclosed by the frame 170 in theexample described above, focus detection signal data obtained under thesecond image-capturing conditions among the sets of focus detectionsignal data available in the area enclosed by the frame 170, may beselected instead.

In the example described above, focus detection is executed byspecifying an area for which the first image-capturing conditions havebeen set and accordingly, focus detection photoelectric conversionsignals, generated under the first image-capturing conditions, areselected for focus detection. A different example in which a focusingtarget subject is located astride an area with the first image-capturingconditions applied thereto and an area with the second image-capturingconditions applied thereto will be described below. When the subject,for which focus is to be adjusted, is located astride an area with thefirst image-capturing conditions applied thereto and an area with thesecond image-capturing conditions applied thereto, the control unit 34(AF operation unit 34 d) selects focus detection photoelectricconversion signals generated under the first image-capturing conditionsas the signals to be used in the focus detection processing, among thefocus detection photoelectric conversion signals generated in the areaenclosed by the frame 170. The control unit 34 (AF operation unit 34 d)then calculates a first defocus quantity based upon the focus detectionphotoelectric conversion signals thus selected. Next, the control unit34 (AF operation unit 34 d) selects focus detection photoelectricconversion signals generated under the second image-capturing conditionsas the signals to be used in the focus detection processing, among thefocus detection photoelectric conversion signals generated in the areaenclosed by the frame 170. The control unit 34 (AF operation unit 34 d)then calculates a second defocus quantity based upon the focus detectionphotoelectric conversion signals thus selected. The control unit 34 (AFoperation unit 34 d) then executes focus detection processing by usingthe first defocus quantity and the second defocus quantity. In morespecific terms, the control unit 34 (AF operation unit 34 d) maycalculate the distance over which the lens needs to move by calculatingthe average value of the first defocus quantity and the second defocusquantity. As an alternative, the control unit 34 (AF operation unit 34d) may select one of the values representing the first defocus quantityand the second defocus quantity that indicates a smaller lens drivedistance. As a further alternative, the control unit 34 (AF operationunit 34 d) may select one of the values representing the first defocusquantity and the second defocus quantity that indicates that the subjectis located closer to the camera.

In addition, when the focusing target subject is present astride an areawith the first image-capturing conditions applied thereto and an areawith the second image-capturing conditions applied thereto, the controlunit 34 (AF operation unit 34 d) may select the focus detectionphotoelectric conversion signals generated in the area occupied by thegreater part of the subject. For instance, 70% of the facial area, i.e.,the focusing target subject, may be present in the area with the firstimage-capturing conditions applied thereto with the remaining 30% of thefacial area in the second area, and in such a case, the control unit 34(AF operation unit 34 d) will select the focus detection photoelectricconversion signals generated under the first image-capturing conditions.It is to be noted that the areal ratios (%) described above simplyrepresents an example, and the present invention is not limited to thisexample.

While an explanation has been given on an example in which the focusdetection processing is executed through the split-pupil phasedifference method, the same principle applies when focus detectionprocessing is executed through the contrast detection method whereby thefocus lens in the image-capturing optical system 31 is driven to thein-focus position based upon variance in the contrast of the subjectimage.

The control unit 34 engaged in focus detection through the contrastdetection method executes focus evaluation value calculation of theknown art, based upon image data output from image-capturing pixels atthe image sensor 32 a, disposed in the area corresponding to the focusdetection position, at each position taken by the focus lens in theimage-capturing optical system 31 as it drives the focus lens. It thendetermines the focus lens position at which the maximum focus evaluationvalue is achieved as the in-focus position.

Under normal circumstances, the control unit 34 executes focusevaluation value calculation by directly utilizing the image data outputfrom the image-capturing pixels disposed in the area corresponding tothe focus detection position without executing any data selectionprocessing. However, if the image data obtained in the areacorresponding to the focus detection position include both image dataobtained under the first image-capturing conditions and image dataobtained under the second image-capturing conditions, the control unit34 selects either the image data obtained under the firstimage-capturing conditions or the image data obtained under the secondimage-capturing conditions, among the sets of image data obtained in thearea corresponding to the focus detection position. The control unit 34then executes the focus evaluation value calculation by using the imagedata selected through the data selection processing. It is to be notedthat when the focusing target subject is present astride an area withthe first image-capturing conditions applied thereto and an area withthe second image-capturing conditions applied thereto, the control unit34 (AF operation unit 34 d) may select the focus detection photoelectricconversion signals generated in the area occupied by the greater part ofthe subject.

3. When Executing Subject Detection Processing

FIG. 14(a) presents an example of a template image 180 of a detectiontarget object, whereas FIG. 14(b) shows a live view image 60(a) and asearch range 190 in an example. The control unit 34 (object detectionunit 34 a) detects a target object (e.g., the purse 63 a, which is oneof the subject elements in FIG. 5) in the live view image. While thecontrol unit 34 (object detection unit 34 a) may set the entire liveview image 60 a as the target object detection range, it may set part ofthe live view image 60 a as a search range 190 so as to lessen thedetection processing load.

If different image-capturing conditions are set for different areas toanother and a boundary between areas is present within the search range190, the control unit 34 (object detection unit 34 a) executes dataselection processing with regard to image data located near the boundarybetween the areas as preliminary processing prior to the subjectdetection processing.

The data selection processing is executed in order to minimize theextent to which the accuracy of the subject element detection processingis lowered due to different image-capturing conditions set for differentareas in the imaging field divided by the setting unit 34 b. Generallyspeaking, when the search range 190 set for purposes of subject elementdetection includes a boundary between areas, image data obtained underdifferent image-capturing conditions may both be present together in theimage data obtained over the search range 190. In the embodiment, dataselection processing is executed as described below based upon thetheory that it is better to detect a subject element by using image dataobtained under uniform image-capturing conditions rather than to detecta subject element by using image data obtained under differentimage-capturing conditions.

An explanation will be given on an example in which the purse 63 a, heldby the person 61 a in the live view image 60 a in FIG. 5 is detected.The control unit 34 (object detection unit 34 a) sets the search range190 in the vicinity of the area containing the person 61 a. It is to benoted that the area 61 in which the person 61 a is present may be simplyset as the search range.

If the search range 190 is not split over two areas with differentimage-capturing conditions applied thereto, the control unit 34 (objectdetection unit 34 a) executes subject detection processing by using theimage data in the search range 190 without executing data selectionprocessing. However, if the image data in the search range 190 includeboth image data obtained under the first image-capturing conditions andimage data obtained under the second image-capturing conditions, thecontrol unit 34 (object detection unit 34 a) selects the image dataobtained under the first image-capturing conditions, among the sets ofimage data in the search range 190, as the image data to be used for thesubject detection processing, as in (example 1) through (example 3)described earlier in reference to the focus detection processing. Thecontrol unit 34 (object detection unit 34 a) then executes subjectdetection processing for the area with the first image-capturingconditions applied thereto by using the image data selected through thedata selection processing. The subject detection processing executed inthis situation may be, for instance, processing through which the targetobject is detected by determining a similarity factor indicating thedegree of similarity between the template image 180 and the selectedimage data corresponding to the first image-capturing conditions (oftenreferred to as template matching). The control unit 34 (object detectionunit 34 a) next selects the image data obtained under the secondimage-capturing conditions, among the sets of image data in the searchrange 190, as the image data to be used in the subject detectionprocessing. The control unit 34 (object detection unit 34 a) executessubject detection processing similar to that described above for thearea with second image-capturing conditions applied thereto by using theimage data thus selected through the data selection processing. Thecontrol unit 34 (object detection unit 34 a) executes subject detectionwithin the search range 190 through this process. Then, it detects thesubject within the search range 190 by combining the subject areadetected by using the image data obtained under the firstimage-capturing conditions and the subject area detected by using theimage data obtained under the second image-capturing conditions at theirboundary. It is to be noted that while the control unit 34 (objectdetection unit 34 a) uses the image data obtained under the firstimage-capturing conditions and the image data obtained under the secondimage-capturing conditions in the example explained above, the subjectmay instead be detected by using image data obtained under either set ofimage-capturing conditions. For instance, while the search range 190includes an area with the first image-capturing conditions appliedthereto and an area with the second image-capturing conditions appliedthereto, the ratio of the area with the first image-capturing conditionsapplied thereto may be greater, and in such a case, subject detectionmay be executed by using the image data obtained under the firstimage-capturing conditions alone. The areal ratio achieved by the areawith the first image-capturing conditions applied thereto may be, forinstance, 70% or more. However, the present invention is not limited tothis example and the areal ratio of the area with the firstimage-capturing conditions applied thereto may be 80% or more or 90% ormore. In other words, it will be obvious that the areal ratio of thearea with the first image-capturing conditions applied thereto may beadjusted to an optimal value.

The data selection processing with regard to image data in the searchrange 190 described above may be executed for a search range set fordetection of a specific subject such as a person's face or for an areaset for purposes of image-capturing scene decision-making. For instance,the control unit 34 (object detection unit 34 a), engaged in detectionof a person's face within the search range 190, selects image dataobtained under the first image-capturing conditions, among the sets ofimage data in the search range 190, as the image data to be used for thesubject detection processing. The control unit 34 (object detection unit34 a) then executes face detection processing of the known art over thearea with the first image-capturing conditions applied thereto. Thecontrol unit 34 (object detection unit 34 a) next selects image dataobtained under the second image-capturing conditions, among the sets ofimage data in the search range 190, as the image data to be used forsubject detection processing. The control unit 34 (object detection unit34 a) executes face detection processing of the known art in the areawith the second image-capturing conditions applied thereto. The controlunit 34 (object detection unit 34 a) then detects a face in the imagedata in the search range 190 by combining the face area detected withinthe area with the first image-capturing conditions applied thereto andthe face area detected within the area with the second image-capturingconditions applied thereto at their boundary.

In addition, data selection similar to that executed with regard to theimage data obtained within the search range 190 described above may beexecuted in conjunction with a search range set for characteristicquantity detection executed based upon image colors, edges and the like,instead of in conjunction with a search range set for pattern matchingexecuted based upon a template image.

Furthermore, template matching processing of the known art may beexecuted by using image data obtained for a plurality of frames obtainedat different time points so as to track a moving object by searching foran area in a frame image obtained at a later time point, which issimilar to a tracking target object in a frame image obtained earlier.In this case, if image data obtained under the first image-capturingconditions and image data obtained under the second image-capturingconditions are both present over the search range set in the frame imageobtained at the later time point, the control unit 34 selects the imagedata obtained under the first image-capturing conditions, among the setsof image data available in the search range, as the image data to beused for tracking processing. The control unit 34 then executes trackingprocessing for the area with the first image-capturing conditionsapplied thereto by using the image data selected through the dataselection processing. Next, the control unit may select image dataobtained under the second image-capturing conditions, among the sets ofimage data available in the search range as image data to be used intracking processing in much the same way as that described above andexecute tracking processing for the area with the second image-capturingconditions applied thereto by using the image data thus selected throughthe data selection processing.

Furthermore, image data corresponding to a plurality of frames obtainedat different time points may be used to detect a motion vector through amethod of the known art. If image data obtained under the firstimage-capturing conditions and image data obtained under the secondimage-capturing conditions are both present within a detection area setfor motion vector detection, the control unit 34 selects the image dataobtained under the first image-capturing conditions among the sets ofimage data in the detection range set for motion vector detection, asthe image data to be used for the detection processing. The control unit34 then detects a motion vector over the area with the firstimage-capturing conditions applied thereto by using the image dataselected through the data selection processing. Subsequently, thecontrol unit 34 may select the image data obtained under the secondimage-capturing conditions, among the sets of image data available inthe search range as the image data to be used for motion vectordetection processing in much the same way as that described above andexecute motion vector detection processing for the area with the secondimage-capturing conditions applied thereto by using the image dataselected through the data selection processing. The control unit 34 maydetermine a motion vector for the entire image based upon the motionvector detected in the area with the first image-capturing conditionsapplied thereto and the motion vector detected in the area with thesecond image-capturing conditions applied thereto, or it may hold themotion vectors detected in the individual areas.

4. When Setting Image-Capturing Conditions

After dividing the imaging field into a plurality of areas and settingdifferent image-capturing conditions for different areas, the controlunit 34 (setting unit 34 b) may execute another photometering operationin order to determine exposure conditions. In such a case, it mayexecute data selection processing with regard to image data located neara boundary of areas as preliminary processing execute before setting theexposure conditions.

The data selection processing is executed in order to minimize theextent to which the accuracy of the processing through which exposureconditions are determined, is lowered due to different image-capturingconditions set in different areas in the imaging field divided by thesetting unit 34 b. For instance, a boundary between areas may be presentwithin a photometering range set at a central area of the imaging fieldand under such circumstances, image data obtained under differentimage-capturing conditions may be included in the image data within thephotometering range. In the embodiment, data selection processing isexecuted as described below based upon the theory that it is better toexecute exposure calculation processing by using image data obtainedunder uniform image-capturing conditions rather than to execute exposurecalculation processing by using image data obtained under differentimage-capturing conditions.

If the photometering range is not split over a plurality of areas withdifferent image-capturing conditions applied thereto, the control unit34 (setting unit 34 b) executes exposure calculation processing by usingthe image data in the photometering range without executing dataselection processing. However, if the image data in the photometeringrange include both image data obtained under the first image-capturingconditions and image data obtained under the second image-capturingconditions, the control unit 34 (setting unit 34 b) selects the imagedata obtained under the first image-capturing conditions, among the setsof image data in the photometering range, as the image data to be usedfor the exposure calculation processing, as in (example 1) through(example 3) described earlier in reference to the focus detectionprocessing and the subject detection processing. The control unit 34(setting unit 34 b) then executes exposure calculation processing in thearea with the first image-capturing conditions applied thereto by usingthe image data selected through the data selection processing. Thecontrol unit 34 (setting unit 34 b) next selects the image data obtainedunder the second image-capturing conditions, among the sets of imagedata in the photometering range, as the image data to be used for theexposure calculation processing. The control unit 34 (setting unit 34 b)executes exposure calculation processing similar to that described abovefor the area with second image-capturing conditions applied thereto byusing the image data thus selected through the data selectionprocessing. Namely, the control unit 34 (setting unit 34 b) executesdata selection processing in order to select data to be used inphotometering the individual areas and executes exposure calculationprocessing by using the image data selected through the data selectionprocessing if a plurality of areas with different image-capturingconditions applied thereto are present within the photometering range.

In addition, if the photometering range is located astride an area withthe first image-capturing conditions applied thereto and an area withthe second image-capturing conditions applied thereto, the control unit34 (setting unit 34 b) may select an area having a greater areal ratio,as has been explained earlier in reference to the focus detection andthe subject detection.

Image data may be selected with regard to a photometering (colorimetryor color-metering) range set when determining a white balance adjustmentvalue, a photometering range set when determining whether or notauxiliary photographing light needs to be emitted from an auxiliaryphotographing light source or a photometering range set when determiningthe amount of auxiliary photographing light to be emitted from the lightsource, instead of the photometering range set for the exposurecalculation processing described above.

In addition, photoelectric conversion signals may be read out at varyingresolutions from one area to another in the imaging field divided intodifferent areas. In such a case, image data may be selected in a similarmanner with regard to an area used to determine the image-capturingscene when setting the readout resolution for each area.

<Flowchart>

FIG. 15 presents a flowchart of the processing executed when capturingan image by setting image-capturing conditions for individual areas. Asa main switch of the camera 1 is turned on, the control unit 34 starts aprogram that enables the processing shown in FIG. 15. In step S10, thecontrol unit 34 engages the display unit 35 to bring up a live viewdisplay, and then the operation proceeds to step S20.

More specifically, the control unit 34 sequentially brings up on displayat the display unit 35 images resulting from predetermined types ofimage processing executed on image data sequentially output from theimage-capturing unit 32, as a live view image. As explained earlier,uniform image-capturing conditions are set at this time for the entirerange of image-capturing chip 111, i.e., over the entire image plane.

It is to be noted that if a setting for executing AF operation while thelive view image display is up is selected, the control unit 34 (AFoperation unit 34 d) controls the AF operation through which focus isadjusted on the subject element corresponding to a specific focusdetection position by executing focus detection processing. The AFoperation unit 34 d executes focus detection processing after executingthe data selection processing described earlier, as needed.

In addition, if the setting for executing AF operation is not selectedwhile the live view display is up, the control unit 34 (AF operationunit 34 d) executes AF operation later in response to an AF operationinstruction.

In step S20, the control unit 34 (object detection unit 34 a) detectssubject elements in the live view image, and then the operation proceedsto step S30. The object detection unit 34 a executes the subjectdetection processing after first executing the data selection processingdescribed earlier, as needed. In step S30, the control unit 34 (settingunit 34 b) divides the imaging field of the live view image into areascontaining the subject elements, before the operation proceeds to stepS40.

In step S40, the control unit 34 displays the areas at the display unit35. As in the example presented in FIG. 6, the control unit 34 displaysthe areas by highlighting the area designated as the image-capturingcondition setting (adjustment) target area. In addition, the controlunit 34 brings up on display at the display unit 35 the image-capturingcondition setting screen 70, before the operation proceeds to step S50.

It is to be noted that if the user taps the display position at whichanother primary subject is displayed on the display screen with hisfinger, the control unit 34 switches the image-capturing conditionsetting (adjustment) target area to the area that includes the otherprimary subject and highlights the new image-capturing condition setting(adjustment) target area on display.

In step S50, the control unit 34 makes a decision as to whether or notAF operation is necessary. The control unit 34 makes an affirmativedecision in step S50 if, for instance, the subject has moved and thefocusing condition has changed, if the focus detection position has beenchanged in response to a user operation or if an instruction for AFoperation execution has been issued through a user operation. Uponmaking an affirmative decision in step S50, the control unit 34 proceedsto step S70. If, on the other hand, the focusing condition has remainedunchanged, the focus detection position has not been changed in responseto a user operation and an instruction for AF operation execution hasnot been issued through a user operation, the control unit 34 makes anegative decision in step S50 and the operation proceeds to step S60.

In step S70, the control unit 34 initiates AF operation, before theoperation returns to step S40. The AF operation unit 34 d executes focusdetection processing for AF operation after executing the data selectionprocessing described earlier as needed. Once the operation has returnedto step S40, the control unit 34 repeatedly executes processing similarto that described above based upon the live view image obtained afterthe AF operation.

In step S60, the control unit 34 (setting unit 34 b) setsimage-capturing conditions for the highlighted area on display inresponse to a user operation and then the operation proceeds to stepS80. It is to be noted that the display at the display unit 35 changesand image-capturing operations are set in response to a user operationin step S60, as has been explained earlier. The control unit 34 (settingunit 34 b) executes exposure calculation processing after executing thedata selection processing described earlier, as needed.

In step S80, the control unit 34 makes a decision as to whether or notan image-capturing instruction has been issued. The control unit 34makes an affirmative decision in step S80 if a shutter release button(not shown) constituting part of the operation member 36 has beenoperated or a display icon via which an image-capturing instruction isissued, has been operated. Upon making an affirmative decision in stepS80, the control unit 34 proceeds to step S90. If no image-capturinginstruction has been issued, the control unit 34 makes a negativedecision in step S80 and the operation returns to step S60.

In step S90, the control unit 34 executes specific image-capturingprocessing. Namely, the image-capturing control unit 34 c controls theimage sensor 32 a so as to capture an image under image-capturingconditions set in correspondence to the individual areas, and then theoperation proceeds to step S100.

In step S100, the control unit 34 (image-capturing control unit 34 c)outputs an instruction to the image processing unit 33 so as to engageit in predetermined types of image processing on the image data obtainedthrough the image-capturing processing, before the operation proceeds tostep S110. The image processing executed in this step includes thedefective pixel correction processing, the color interpolationprocessing, the edge enhancement processing and the noise reductionprocessing explained earlier.

It is to be noted that the image processing unit 33 (selection unit 33b) executes the image processing after executing data selectionprocessing for image data located near a boundary of areas, as needed.

In step S110, the control unit 34 outputs an instruction to therecording unit 37 so as to record the image data having undergone theimage processing into a recording medium (not shown), and then theoperation proceeds to step S120.

In step S120, the control unit 34 makes a decision as to whether or notan end operation has been performed. If an end operation has beenperformed, the control unit 34 makes an affirmative decision in stepS120 and ends the processing in FIG. 15. If, on the other hand no endoperation has been performed, the control unit 34 makes a negativedecision in step S120 and the operation returns to step S20. Once theoperation has returned to step S20, the control unit 34 repeatedlyexecutes the processing described above.

While the image sensor 32 a in the example described above is a stackedimage sensor 100, it is not necessary for the image sensor to beconfigured as a stacked image sensor, as long as it allows differentimage-capturing conditions to be set from one block to another among aplurality of blocks at the image sensor (image-capturing chip 111).

The following advantages and operations are achieved through theembodiment inscribed above.

(1) The camera 1, equipped with an image processing device, includes animage processing unit 33 (input unit 33 a), to which first image dataobtained by capturing a subject image having entered a first area at animage-capturing unit 32, under first image-capturing conditions, andsecond image data obtained by capturing a subject image having entered asecond area at the image-capturing unit 32, under second image-capturingconditions different from the first image-capturing conditions, areinput, a selection unit 33 b that selects either the first image data orthe second image data input to the input unit 33 a, and an imageprocessing unit (generation unit 33) that brings up on display at thedisplay unit 35 an image generated by using the selected image data.This structure makes it possible to execute the optimal processingindividually in areas with different image-capturing conditions appliedthereto. In other words, an image can be generated in an optimal mannerby using the sets of image data obtained in the individual areas. Forinstance, the unnatural appearance that would otherwise occur in animage, due to different image-capturing conditions being set fordifferent areas, can be minimized.

(2) The generation unit 33 c in the camera 1 generates an image by usingimage data available at specific reference pixels located around eachtarget pixel taking a specific position in an image that includes thefirst image data and the second image data, whereas the selection unit33 b selects either the first image data or the second image data as theimage data at the reference pixels in correspondence to the positiontaken by each target pixel. Through these measures, optimal processingcan be executed individually for areas with different image-capturingconditions applied thereto.

(3) The selection unit 33 b in the camera 1 selects either the firstimage data or the second image data obtained by capturing an image underimage-capturing conditions matching the image-capturing conditions setfor the target pixel. Through these measures, optimal processing can beexecuted individually for areas with different image-capturingconditions applied thereto.

(4) The selection unit 33 b in the camera 1 also selects image dataobtained at another pixel, different from the reference pixels, underimage-capturing conditions matching those set for the target pixel.Through these measures, optimal processing can be executed individuallyfor areas with different image-capturing conditions applied thereto.

(5) If the entire image data are made up exclusively with first imagedata, the selection unit 33 b in the camera 1 selects all the imagedata. If the entire image data are made up exclusively with second imagedata, it selects all the image data. If the image data include bothfirst image data and second image data, it selects either the firstimage data or the second image data. Through these measures, optimalprocessing can be executed individually for areas with differentimage-capturing conditions applied thereto.

(6) Since the first image-capturing conditions and the secondimage-capturing conditions set at the camera 1 include at least eitherthe accumulation time or the ISO sensitivity, optimal processing can beexecuted in areas with different electric charge accumulation timesettings or different image-capturing sensitivity settings appliedthereto.

(7) The generation unit 33 c in the camera 1 generates third image datathrough at least defective pixel correction processing, colorinterpolation processing, edge enhancement processing or noise reductionprocessing. As a result, image data having undergone defective pixelcorrection processing, color interpolation processing, edge enhancementprocessing or noise reduction processing optimally executed for areaswith different image-capturing conditions applied thereto, can begenerated.

Second Embodiment

In the image data selection processing executed in conjunction withimage processing in the first embodiment, if image-capturing conditions(referred to as first image-capturing conditions) applied at a targetpixel are different from image-capturing conditions (referred to assecond image-capturing conditions) applied at reference pixels presentaround the target pixel P, the image processing unit 33 (selection unit33 b) selects image data obtained under the first image-capturingconditions matching those applied at the target pixel, among the sets ofimage data available at pixels located within a predetermined range 90,and the image processing unit 33 (generation unit 33 c) references theselected image data.

In the second embodiment, the image processing unit 33 (selection unit33 b) also selects image data obtained under the first image-capturingconditions matching those applied at the target pixel, from sets ofimage data available at pixels located outside the predetermined range90, as well, so as to allow the image processing unit 33 (generationunit 33 c) to reference a greater number of sets of data. Namely, theimage processing unit 33 (selection unit 33 b) adjusts selection rangeand selects image data obtained under the first image-capturingconditions.

FIG. 7(d) provides an enlarged view of a target pixel P and referencepixels designated in the second embodiment. In FIG. 7(d), the dataoutput from the unshaded pixels are image data obtained under the firstimage-capturing conditions, whereas the data output from the shadedpixels are image data obtained under the second image-capturingconditions within the predetermined range 90 centered on the targetpixel P. As in the first embodiment, the image processing unit 33(selection unit 33 b) does not select image data available at the pixels(shaded pixels) with the second image-capturing conditions appliedthereto even though they are located within the predetermined range 90.

The image processing unit 33 (selection unit 33 b) in the secondembodiment further selects image data at the unshaded pixels locatedoutside the predetermined range 90 with the first image-capturingconditions applied thereto, as well as the image data available at theunshaded pixels located within the predetermined range 90 with the firstimage-capturing conditions applied thereto, as illustrated in FIG. 7(d).The image processing unit 33 (generation unit 33 c) executes imageprocessing by referencing the image data thus selected. In the exampledescribed above, the distances from the position taken by the targetpixel P to the positions at which the image data have been obtainedunder the second image-capturing conditions, located inside thepredetermined range 90, are greater than the distances from the positionof the target pixel P to the positions at which the image data obtainedat the unshaded pixels located outside the predetermined range 90 withthe first image-capturing conditions applied thereto. In other words,the image processing unit 33 (generation unit 33 c) is allowed to selectimage data obtained under the first image-capturing conditions availableat pixels set apart from the target pixel P by greater distances ratherthan select image data obtained under the second image-capturingconditions at pixels set apart from the target pixel P by smallerdistances.

It is to be noted that when selecting image data, the image processingunit 33 (selection unit 33 b) gives priority to image data available ata pixel taking a position closer to the predetermined range 90 overimage data available at a pixel taking a position further distanced fromthe predetermined range 90. The rationale for this approach is that apixel located closer to the predetermined range 90 is more likely tohold image information commonly shared with the target pixel P than apixel located further away from the predetermined range 90.

In addition, with L representing the length of a side of thepredetermined range 90, the image processing unit 33 (selection unit 33b) selects image data available at an unshaded pixel (i.e., with thefirst image-capturing conditions applied thereto) set apart from thetarget pixel by a distance, indicated by a diagonal line, equal to L orless. The rationale for this selection approach is that it is desirableto not include image data at a pixel set apart from the predeterminedrange 90 by too great a distance since such a pixel is less likely tohold image information commonly shared with the target pixel P.

<Examples of Image Processing>

Examples of image processing that may be executed in the secondembodiment will be described.

(1) Defective Pixel Correction Processing

If uniform image-capturing conditions have been applied at the targetpixel P and all the pixels present within the predetermined range 90centered on the target pixel P, the image processing unit 33 (selectionunit 33 b) selects all the sets of image data at the pixels locatedinside the predetermined range 90. Then, the image processing unit 33(generation unit 33 c) executes Max, Min filter processing byreferencing the selected image data. It is to be noted that it mayinstead execute defective pixel correction processing by taking theaverage of the selected image data.

As illustrated in FIG. 7(d), if any pixel, for which the secondimage-capturing conditions, different from the first image-capturingconditions applied at the target pixel P have been set, is present inthe predetermined range 90 centered on the target pixel P, the imageprocessing unit 33 (selection unit 33 b) selects the image data at thepixels located within the predetermined range 90 with the firstimage-capturing conditions applied thereto. It also selects the imagedata at the unshaded pixels, located outside the predetermined range 90with the first image-capturing conditions applied thereto. The imageprocessing unit 33 (generation unit 33 c) then executes the Max, Minprocessing described above by referencing the image data thus selected.It is to be noted that it may instead execute defective pixel correctionprocessing by taking the average of the selected image data.

The image processing unit 33 executes the processing described above forall the defective pixels, position information for which is recorded inthe non-volatile memory.

(2) Color Interpolation Processing

<G Color Interpolation>

Next, the G color interpolation executed in the second embodiment willbe explained. In FIG. 8(a) through FIG. 8(c), the first image-capturingconditions are set for the area located to the left of and above thebold lines and the second image-capturing conditions are set for thearea located to the right of and below the bold lines, as in the firstembodiment.

In the example presented in FIG. 8(b), the second image-capturingconditions, different from the first image-capturing conditions appliedat the target position (second row/second column) are applied at thereference position corresponding to the shaded G-color component imagedata G4. Accordingly, the image processing unit 33 (selection unit 33 b)selects the image data G1 through G3 obtained under the firstimage-capturing conditions among the sets of G color component imagedata G1 through G4. In addition, the image processing unit 33 (selectionunit 33 b) selects G color component image data G6 obtained under thefirst image-capturing conditions, located near the reference positioncorresponding to the data G4. In other words, the current embodiment isdistinguishable from the first embodiment in that the image processingunit 33 (selection unit 33 b) selects image data obtained under thefirst image-capturing conditions by setting alternative image dataselection positions.

It is to be noted that if the second image-capturing conditions are alsoapplied at the position corresponding to the data G6, data obtainedunder the first image-capturing conditions, among the sets of image dataavailable at positions near the data G6, may be selected.

The image processing unit 33 (generation unit 33 c) executes calculationso as to generate G color component image data for the target position(second row/second column) by referencing the image data selected asdescribed above. The image processing unit 33 (generation unit 33 c)may, for instance, generate G color component image data(a2G1+b2G2+c2G3+d2G6)/4 for the target position (second row/secondcolumn). It is to be noted that a2, b2, c2 and d2 are weightingcoefficients each selected in correspondence to the distance between thespecific reference position and the target position and the imagestructure.

The image processing unit 33 (generation unit 33 c) is able to obtain Gcolor component image data for each pixel position, as shown in FIG.8(c) by generating G color component image data at each position amongthe B color component positions and the R color component positions inFIG. 8(a).

<R Color Interpolation>

The R color interpolation executed in the second embodiment will beexplained. In FIG. 9(a) through FIG. 9(c) the first image-capturingconditions are set for the area located to the left of and above thebold lines and the second image-capturing conditions are set for thearea located to the right of and below the bold lines, as in the firstembodiment.

In the example presented in FIG. 9(b), the second image-capturingconditions, different from the first image-capturing conditions appliedat the target position indicated with the bold-line frame (secondrow/second column) are applied at the reference position correspondingto the shaded image data Cr2 representing the chrominance component Cr.Accordingly, the image processing unit 33 (selection unit 33 b) selectsthe chrominance component image data Cr1 and Cr3 through Cr4 obtainedunder the first image-capturing conditions among the chrominancecomponent image data Cr1 through Cr4. In addition, the image processingunit 33 (selection unit 33 b) selects image data Cr15 (or Cr16) for thechrominance component Cr obtained under the first image-capturingconditions, located near the reference position corresponding to thedata Cr2. In other words, the current embodiment can be distinguishedfrom the first embodiment in that the image processing unit 33(selection unit 33 b) selects image data obtained under the firstimage-capturing conditions by setting an alternative image dataselection position.

It is to be noted that if the second image-capturing conditions are alsoapplied at the position corresponding to the data Cr15 or Cr16, dataobtained under the first image-capturing conditions, among the sets ofimage data available at positions near the data Cr15 or Cr16 may beselected.

The image processing unit 33 (generation unit 33 c) executes calculationso as to generate chrominance component image data for the targetposition (second row/second column) by referencing the image dataselected as described above. The image processing unit 33 (generationunit 33 c) may, for instance, generate chrominance component image dataCr (e3Cr1+f3Cr15+g3Cr3+h3Cr4)/4 for the target position (secondrow/second column). It is to be noted that e3, f3, g3 and h3 areweighting coefficients each selected in correspondence to the distancebetween the specific reference position and the target position and theimage structure.

Chrominance component image data Cr are generated for the targetposition indicated with the bold-line frame (second row/third column) inFIG. 9(c) in much the same manner. In the example presented in FIG.9(c), the first image-capturing conditions, different from the secondimage-capturing conditions applied at the target position indicated withthe bold-line frame (second row/third column) are applied at thereference positions corresponding to the shaded image data Cr4 and Cr5representing the chrominance component Cr. Accordingly, the imageprocessing unit 33 (selection unit 33 b) selects the chrominancecomponent image data Cr2 and Cr6 obtained under the secondimage-capturing conditions among the chrominance component image dataCr2 and Cr4 through Cr6. In addition, the image processing unit 33(selection unit 33 b) selects chrominance component image data Cr8 andCr7 obtained under the second image-capturing conditions, located nearthe reference position corresponding to the data Cr4 and

Cr5. In other words, the current embodiment can be distinguished fromthe first embodiment in that the image processing unit 33 (selectionunit 33 b) selects image data obtained under the second image-capturingconditions by setting alternative image data selection positions.

It is to be noted that if the first image-capturing conditions are alsoapplied at the position corresponding to the data Cr8 and Cr7, dataobtained under the second image-capturing conditions, among the sets ofimage data available at positions near the data Cr8 and Cr7 may beselected.

The image processing unit 33 (generation unit 33 c) executes calculationso as to generate chrominance component image data Cr for the targetposition (second row/third column) by referencing the image dataselected as described above. The image processing unit 33 (generationunit 33 c) may generate chrominance component image data Cr(q3Cr2+r3Cr8+sCr7+t3Cr6)/4 for the target position. It is to be notedthat q3, r3, s3 and t3 are weighting coefficients each selected incorrespondence to the distance between the specific reference positionand the target position and the image structure.

Once chrominance component image data Cr have been obtained for eachpixel position, the image processing unit 33 (generation unit 33 c) isable to obtain R color component image data at each pixel position byadding each set of G color component image data in FIG. 8(c) to thechrominance component image data Cr at the corresponding pixel position.

<B Color Interpolation>

Next, the B color interpolation executed in the second embodiment willbe explained. In FIG. 10(a) through FIG. 10(c) the first image-capturingconditions are set for the area located to the left of and above thebold lines and the second image-capturing conditions are set for thearea located to the right of and below the bold lines.

In the example presented in FIG. 10(b), the first image-capturingconditions, different from the second image-capturing conditions appliedat the target position indicated with the bold-line frame (thirdrow/third column) are applied at the reference positions correspondingto the shaded image data Cb1 and Cb3 representing the chrominancecomponent Cb. Accordingly, the image processing unit 33 (selection unit33 b) selects the chrominance component image data Cb Cb2 and Cb4obtained under the second image-capturing conditions among the sets ofchrominance component image data Cb1 through Cb4. In addition, the imageprocessing unit 33 (selection unit 33 b) selects image data Cb16 andCb17 for the chrominance component Cb obtained under the secondimage-capturing conditions, located near the reference positioncorresponding to the data Cb1 and Cb3. In other words, the currentembodiment can be distinguished from the first embodiment in that theimage processing unit 33 (selection unit 33 b) selects image dataobtained under the second image-capturing conditions by settingalternative image data selection positions.

The image processing unit 33 (generation unit 33 c) executes calculationso as to generate chrominance component image data Cb for the targetposition (third row/third column) by referencing the image data selectedas described above. The image processing unit 33 (generation unit 33 c)may, for instance, generate chrominance component image data Cb(u3Cb16+v3Cb2+w3Cb4+x3Cb17)/4 for the target position (third row/thirdcolumn). It is to be noted that u3, v3, w3 and x3 are weightingcoefficients each selected in correspondence to the distance between thespecific reference position and the target position and the imagestructure.

Chrominance component image data Cb are generated for the targetposition are indicated with the bold-line frame (third row/fourthcolumn) in FIG. 10(c) in much the same manner.

It is to be noted that the second image-capturing conditions matchingthose applied at the target position (third row/fourth column) are setat the reference positions corresponding to the four sets of chrominancecomponent image data Cb2 and Cb4 through Cb6, located near the targetposition (third row/fourth column) in the example presented in FIG.10(c). The image processing unit 33 (generation unit 33 c) generatesthrough calculation chrominance component image data Cb at the targetposition by referencing the four sets of chrominance component imagedata Cb2 and Cb4 through Cb6, located near the target position.

Once chrominance component image data Cb have been generated for eachpixel position, the image processing unit 33 (generation unit 33 c) isable to obtain B color component image data at each pixel position byadding each set of G color component image data in FIG. 8(c) to thechrominance component image data Cr at the corresponding pixel position.

Through the second embodiment described above, the following advantageand operation, in addition to the advantages and operations similar tothose achieved in the first embodiment, are realized. Namely, theselection unit 33 b selects image data at a pixel that is not areference pixel, present in an area for which image-capturing conditionsidentical to those applied at the pixel position (the position taken bythe target pixel) have been set. As a result, image processing can beexecuted in an optimal manner by, for instance, increasing the number ofsets of data referenced by the generation unit 33 c while it is engagedin generation of third image data.

Variations of the First and Second Embodiments

The following variations are also within the scope of the presentinvention, and one of the variations or a plurality of variations may beadopted in combination with either of the embodiments described above.

(Variation 1) FIG. 16(a) through FIG. 16(c) each present an example of apositional arrangement with which a first area and a second area may beset on the image-capturing surface of the image sensor 32 a. In theexample presented in FIG. 16(a), the first area is made up witheven-numbered columns and the second area is made up with odd-numberedcolumns. Namely, the image-capturing surface is divided intoeven-numbered columns and odd-numbered columns.

In the example presented in FIG. 16(b), the first area is made up withodd-numbered rows and the second area is made up with even-numberedrows. Namely, the image-capturing surface is divided into odd-numberedrows and even-numbered rows.

In the example presented in FIG. 16(c), the first area is made up witheven-numbered row blocks in odd-numbered columns and odd-numbered rowblocks in even-numbered columns. In addition, the second area is made upwith even-numbered row blocks in even-numbered columns and odd-numberedrow blocks in odd numbered columns. Namely, the image-capturing surfaceis divided into areas forming a checker pattern.

In each of the examples presented in FIG. 16(a) through FIG. 16(c),photoelectric conversion signals read out from the image sensor 32 ahaving captured an image for one frame are used to generate a firstimage based upon photoelectric conversion signals read out from thefirst area and a second image based upon photoelectric conversionsignals read out from the second area. In Variation 1, the first imageand the second image are captured at matching angles of view and includea common subject image.

In Variation 1, the control unit 34 uses the first image as a displayimage and uses the second image for purposes of detection. In morespecific terms, the control unit 34 brings up on display at the displayunit 35 the first image as a live view image. In addition, the controlunit 34 engages the object detection unit 34 a in subject detectionprocessing executed by using the second image, engages the AF operationunit 34 in focus detection processing by using the second image andengages the setting unit 34 b in exposure calculation processingexecuted by using the second image.

In the description of Variation 1, the image-capturing conditions setfor the first area where the first image is captured are referred to asfirst image-capturing conditions and the image-capturing conditions setfor the second area where the second image is captured are referred toas second image-capturing conditions. The control unit 34 may selectdifferent image-capturing conditions for the first image-capturingconditions and the second image-capturing conditions.

1. For instance, the control unit 34 may select conditions optimal fordisplay at the display unit 35 as the first image-capturing conditions.In this situation, uniform image-capturing conditions will be set as thefirst image-capturing conditions for the entire first area at theimaging field. At the same time, the control unit 34 may selectconditions optimal for focus detection processing, subject detectionprocessing and exposure calculation processing as the secondimage-capturing conditions. Uniform image-capturing conditions will bealso be set as the second image-capturing conditions for the entiresecond area at the imaging field.

It is to be noted that if conditions optimal for the focus detectionprocessing, conditions optimal for the subject detection processing andconditions optimal for the exposure calculation processing are differentfrom one another, the control unit 34 may select different secondimage-capturing conditions for the second area from one frame to thenext. For instance, it may select the conditions optimal for the focusdetection processing as the second image-capturing conditions for thefirst frame, select the conditions optimal for the subject detectionprocessing as the second image-capturing conditions for the secondframe, and select the conditions optimal for the exposure calculationprocessing as the second image-capturing conditions for the third frame.In this case too, uniform image-capturing conditions will be set for thesecond image-capturing conditions for the entire second area at theimaging field in correspondence to each frame.

2. As an alternative, the control unit 34 may select differentimage-capturing conditions as the first image-capturing conditions setfor the first area. In this case, the control unit 34 (setting unit 34b) sets different first image-capturing conditions from one subjectelement-containing area to another among areas containing differentsubject elements defined by the setting unit 34 b. However, the controlunit 34 selects uniform second image-capturing conditions for the entiresecond area at the imaging field. While the control unit 34 selectsconditions optimal for focus detection processing, subject detectionprocessing and exposure calculation processing as the secondimage-capturing conditions, it may select different image-capturingconditions for the second area from one frame to another if conditionsoptimal for the focus detection processing, conditions optimal for thesubject detection processing and conditions optimal for the exposurecalculation processing are different from one another.

3. As a further alternative, the control unit 34 may select differentsecond image-capturing conditions for the second area at the imagingfield while applying uniform first image-capturing conditions to theentire first area of the imaging field. For instance, it may selectdifferent second image-capturing conditions from one subjectelement-containing area to another among areas containing differentsubject elements defined by the setting unit 34. In this case, too,different image-capturing conditions may be set for the second area fromone frame to another if conditions optimal for the focus detectionprocessing, conditions optimal for the subject detection processing andconditions optimal for the exposure calculation processing are differentfrom one another.

4. As yet another alternative, the control unit 34 may select differentfirst image-capturing conditions for the first area at the imaging fieldand also select different second image-capturing conditions for thesecond area at the imaging field. For instance, it may select differentfirst image-capturing conditions from one subject element-containingarea to another among areas containing different subject elementsdefined by the setting unit 34 b and also select different secondimage-capturing conditions from one subject element-containing area toanother among areas containing different subject elements defined by thesetting unit 34 b.

The areal ratios of the first area and the second area in FIG. 16(a)through FIG. 16(c) may be varied. In response to a user operation orbased upon a decision made by the control unit 34, for instance, thecontrol unit 34 may set the ratio of the first area higher than that ofthe second area may set the first area ratio and the second area ratioequal to each other, as shown in the examples presented in FIG. 16(a)through FIG. 16(c) or may set the ratio of the first area lower thanthat of the second area. By adjusting the ratios for the first area andthe second area, a first image achieving higher definition compared tothe second image can be generated, the resolution of the first image andthe resolution of the second image can be set equal to each other or asecond image achieving higher definition than the first image can begenerated.

(Variation 2)

In the embodiments described earlier, the control unit 34 (setting unit34 b) detects subject elements based upon the live view image anddivides the image plane of the live view image into subjectelement-containing areas. In conjunction with a photometering sensorinstalled as a member separate from the image sensor 32 a, the controlunit 34 in Variation 2 may divide the live view image plane intodifferent areas based upon output signals provided by the photometeringsensor.

In this case, based upon output signals from the photometering sensor,the control unit 34 divides the image plane into a foreground and abackground. In more specific terms, it divides the live view imageobtained via the image sensor 32 b into a foreground area correspondingto an area judged to be a foreground based upon signals output from thephotometering sensor and a background area corresponding to an areajudged to be a background based upon signals output from thephotometering sensor.

The control unit 34 then sets a first area and a second area atpositions corresponding to the foreground area at the image-capturingsurface of the image sensor 32 a, as in any of the examples presented inFIG. 16(a) through FIG. 16(c). In addition, the control unit 34 sets thefirst area alone at positions corresponding to the background area atthe image-capturing surface of the image sensor 32 a. The control unit34 uses the first image as a display image and uses the second image forpurposes of detection.

In Variation 2, the live view image obtained via the image sensor 32 bcan be divided into different areas by using output signals providedfrom a photometering sensor. In addition, a first image to be used as adisplay image and a second image to be used for purposes of detectioncan be obtained in correspondence to the foreground area, whereas afirst image to be used as a display image alone can be obtained over thebackground area.

(Variation 3)

In Variation 3, the image processing unit 33 (generation unit 33 c)executes contrast adjustment processing in order to reduce theappearance of discontinuity in an image, attributable to the differencebetween the first image-capturing conditions and the secondimage-capturing conditions. Namely, the generation unit 33 c lessens theappearance of discontinuity in an image attributable to the differencebetween the first image-capturing conditions and the secondimage-capturing conditions by altering the tone curve (gamma curve).

For instance, the first image-capturing conditions and the secondimage-capturing conditions may differ from each other only in the ISOsensitivity, with an ISO sensitivity setting of 100 selected for thefirst image-capturing conditions and an ISO sensitivity setting of 800selected for the second image-capturing conditions. In this situation,the generation unit 33 c will compress the image data values obtainedunder the second image-capturing conditions, among the image data valuesindicated at the reference positions by a factor of ⅛ by flattening thetone curve.

As an alternative, the generation unit 33 c may expand the image datavalue included at the target position and image data values obtainedunder the first image-capturing conditions, among the image data valuesindicated at the reference positions by a factor of eight, by steepeningthe tone curve.

As in the embodiments described earlier, optimal image processing can beexecuted on sets of image data individually generated in correspondenceto areas with different image-capturing conditions applied theretothrough Variation 3. For instance, the appearance of discontinuity or anunnatural appearance in an image having undergone image processing,attributable to different image-capturing conditions set at boundariesof areas can be reduced.

(Variation 4)

In Variation 4, the image processing unit 33 (generation unit 33 c)executes the image processing described above (e.g., noise reductionprocessing) by ensuring that edges of subject elements do not becomelost. Noise reduction is generally executed through smoothing filterprocessing. While a smoothing filter is effective in noise reduction,the use of a smoothing filter may result in blurring at the edges ofsubject elements.

Accordingly, the image processing unit 33 (generation unit 33 c)corrects any blurring at the edges of the subject elements by executingcontrast adjustment processing, for instance, after the noise reductionprocessing or concurrently while the noise reduction processing isunderway. The image processing unit 33 (generation unit 33 c) inVariation 4 sets an S-shaped curve as a density conversion (gradationconversion) curve (for conversion often referred to as S conversion).The image processing unit 33 (generation unit 33 c) executes contrastadjustment through the S conversion so as to increase the number ofgradations in bright-area image data and the number of gradations indark-area image data by individually stretching the bright datagradation portion and the dark data gradation portion and to decreasethe number of gradations for mid tone gradation image data bycompressing the mid-tone gradation image data. Through these measures,blurring that occurs along the edges of subject elements can becorrected as a result of a decrease in the number of sets of image dataassuming mid-tone image brightness and an increase in the number of setsof image data classified as either bright-area data or dark-area data.

Through Variation 4, by achieving sharper contrast between bright anddark in the image, blurring along the boundaries of subject elements canbe corrected.

(Variation 5)

In Variation 5, the image processing unit 33 (generation unit 33 c)adjusts white balance adjustment gain to reduce the appearance ofdiscontinuity in an image attributable to the difference between thefirst image-capturing conditions and the second image-capturingconditions.

For instance, the image-capturing conditions (referred to as firstimage-capturing conditions) applied at the target position duringimage-capturing operation and the image-capturing conditions (referredto as second image-capturing conditions) applied at a reference positionaround the target position during the image-capturing operation may bedifferent from each other, and under such circumstances, the imageprocessing unit 33 (generation unit 33 c) adjusts the white balanceadjustment gain so that the white balance in the image data obtainedunder the second image-capturing conditions, among the sets of imagedata obtained at reference positions, will achieve better similarity tothe white balance in the image data obtained under the firstimage-capturing conditions.

It is to be noted that the image processing unit 33 (generation unit 33c) may instead adjust the white balance adjustment gain so as to adjustthe white balance in the image data obtained under the firstimage-capturing conditions, among the sets of image data obtained at thereference positions, and in the image data obtained at the targetposition to achieve better similarity to the white balance in the imagedata obtained under the second image-capturing conditions.

In Variation 5, the appearance of discontinuity in an image,attributable to the difference between the first image-capturingconditions and the second image-capturing conditions, can be reduced byadjusting the white balance adjustment gains corresponding to the imagedata obtained in areas at which different image-capturing conditions areapplied so that a uniform adjustment gain matching the gain applied ineither of the areas with different image-capturing conditions appliedthereto is used for both areas.

(Variation 6)

A plurality of image processing units 33 may be used for concurrentexecution of image processing. For instance, while image processing forimage data obtained by capturing an image in an area A at theimage-capturing unit 32 is underway, image processing may be alsoexecuted for image data obtained by capturing an image in an area B atthe image-capturing unit 32. The plurality of image processing units 33may execute identical image processing or they may be engaged indifferent types of image processing. Namely, matching image processingmay be executed for the image data obtained in the area A and the imagedata obtained in the area B by using matching parameters or the like, ordifferent types of image processing may be executed for the image dataobtained in the area A and for the image data obtained in the area B byusing different parameters.

In a configuration that includes a plurality of image processing units133, one of the image processing units may be engaged in imageprocessing for image data obtained under the first image-capturingconditions and another image processing unit may be engaged in imageprocessing for image data obtained under the second image-capturingconditions. In addition, the number of image processing units is notlimited to two, as in the example described above, and the number ofimage processing units included in the configuration may match thenumber of sets of image-capturing conditions that can be set. Namely, animage processing unit may be assigned to execute image processing foreach area for which a different set of image-capturing conditions hasbeen set. In Variation 6, image-capturing operation executed underdifferent image-capturing conditions set for different areas and imageprocessing for image data expressing an image obtained in correspondenceto each of the areas can be executed concurrently.

(Variation 7)

While the explanation has been given on an example in which the presentinvention is embodied in the camera 1, the present invention is notlimited to this example and it may be adopted in a high-function mobiletelephone 250 (see FIG. 14) with a camera function such as a smartphone, or in a mobile device such as a tablet terminal.

(Variation 8)

In the embodiments described above, the present invention is adopted inthe camera 1, i.e., an integrated electronic device having theimage-capturing unit 32 and the control unit 34 included as integratedcomponents thereof. As an alternative, the present invention may beadopted in an image-capturing system 1B configured with animage-capturing unit 32 and a control unit 34, provided as entitiesseparate from each other, in which the image-capturing unit 32 iscontrolled by the control unit 34 through communication.

The following is a description of an example in which an image-capturingdevice 1001, which includes an image-capturing unit 32, is controlled bya control device 1002 which includes a control unit 34, given inreference to FIG. 17.

FIG. 17 is a block diagram presenting an example of a structure that maybe adopted in the image-capturing system 1B in Variation 8. Theimage-capturing system 1B in FIG. 17 is configured with theimage-capturing device 1001 and a display device 1002. Theimage-capturing device 1001 includes a first communication unit 1003, aswell as the image-capturing optical system 31 and the image-capturingunit 32 described in reference to the embodiments. In addition, thedisplay device 1002 includes a second communication unit 1004, as wellas the image processing unit 33, the control unit 34, the display unit35, the operation member 36 and the recording unit 37 described inreference to the embodiments.

The first communication unit 1003 and the second communication unit 1004are able to conduct bidirectional image data communication through, forinstance, wireless communication, optical communication or the like ofthe known art.

It is to be noted that the image-capturing device 1001 and the displaydevice 1002 may be connected via a wiring cable to enable the firstcommunication unit 1003 and the second communication unit 1004 toconduct bidirectional image data communication.

In the image-capturing system 1B, the control unit 34 controls theimage-capturing unit 32 through data communication enabled via thesecond communication unit 1004 and the first communication unit 1003.For instance, specific image data are exchanged between theimage-capturing device 1001 and the display device 1002 so as to allowthe display device 1002 to divide the image plane into a plurality ofareas based upon an image, set different image-capturing conditions indifferent areas and read out photoelectric conversion signals resultingfrom photoelectric conversion in the individual areas, as has beenexplained earlier.

In Variation 8, a live view image obtained at the image-capturing device1001 and transmitted to the display device 1002 is brought up on displayat the display unit 35 in the display device 1002. This means that theuser is able to perform operation through remote control at the displaydevice 1002 distanced from the image-capturing device 1001.

The display device 1002 may be, for instance, a high-function mobiletelephone 250 such as a smart phone. In addition, the image-capturingdevice 1001 may be configured as an electronic device that includes thestacked image sensor 100 described earlier.

It is to be noted that while the control unit 34 in the display device1002 includes the object detection unit 34 a, the setting unit 34 b, theimage-capturing control unit 34 c and the AF operation unit 34 d in theexample described above, some of these units, i.e., the subjectdetection unit 34 a, the setting unit 34 b, the image-capturing controlunit 34 c and the AF operation unit 34 d, may be installed in theimage-capturing device 1001, instead.

(Variation 9)

A program may be provided to the camera 1, the high-function mobiletelephone 250 or a mobile device such as a tablet terminal bytransmitting the program installed in the personal computer 205 to therecipient mobile device through, for instance, infrared communication orshort-range wireless communication, as illustrated in FIG. 18.

The program may be provided to the personal computer 205 by setting arecording medium 204, such as a CD-ROM, with the program stored therein,into the personal computer 205, or by loading the program into thepersonal computer 205 via a communication line 201 such as a network.The program to be provided via the communication line 201 should bestored in a storage device 203 or the like at a server 202 connected tothe communication line.

In addition, the program may be directly transmitted to the mobiledevice via an access point (not shown) of a wireless LAN connected tothe communication line 201. Furthermore, a recording medium 204B such asa memory card with the program stored therein may be set in the mobiledevice. Namely, the program may be provided as a computer programproduct assuming any of various modes, including a computer programproduct in a recording medium and a computer program product providedvia a communication line.

Third Embodiment

In reference to FIGS. 19 through 25, a digital camera representing anexample of an electronic device that includes the image processingdevice achieved in the third embodiment, will be explained. Thefollowing explanation will focus on features distinguishing theembodiment from the first and second embodiments with the same referencesigns assigned to structural elements identical to those in the first orsecond embodiment. Elements that are not specially noted are identicalto those in the first or second embodiment. The primary featuredistinguishing the embodiment from the first embodiment is that thestructure in the embodiment does not include the image processing unit33 in the first embodiment and instead an image-capturing unit 32Afurther includes an image processing unit 32 c having functions similarto those of the image-capturing unit 33 in the first embodiment.

FIG. 19 is a block diagram presenting an example of a structure that maybe adopted in a camera 1C in the third embodiment. The camera 1C in FIG.19 comprises an image-capturing optical system 31, the image-capturingunit 32A, a control unit 34, a display unit 35, an operation member 36and a recording unit 37. The image-capturing unit 32A further includesan image processing unit 32 c having functions similar to those of theimage processing unit 33 in the first embodiment.

The image processing unit 32 c includes an input unit 321, selectionunits 322 and a generation unit 323. Image data obtained via an imagesensor 32 a are input to the input unit 321. The selection units 322execute preliminary processing on the image data input as describedabove. The preliminary processing executed by the selection units 322 isidentical to the preliminary processing executed by the selection unit33 b in the first embodiment. The generation unit 323 generates an imageby executing image processing on the input image data and the image datahaving undergone the preliminary processing.

The image processing executed by the generation unit 323 is identical tothe image processing executed by the generation unit 33 c in the firstembodiment.

FIG. 20 schematically illustrates the corresponding relationship thatexists between various blocks and a plurality of selection units 322 inthe embodiment. In FIG. 20 each square at a rectangular image-capturingchip 111 represents a single block 111 a. Likewise, each square at arectangular image processing chip 114, to be described later, representsa single selection unit 322.

In the embodiment, the selection units 322 are provided each incorrespondence to one of the blocks 111 a. In other words, a selectionunit 322 is provided for each of the blocks which are the smallest unitsof areas for which different image-capturing conditions can be set atthe imaging plane. For instance, the hatched block 111 a and the hatchedselection unit in FIG. 20 correspond to each other. The hatchedselection unit 322 in FIG. 20 executes the preliminary processing forthe image data output from the pixels present in the hatched block 111a. The selection units 322 each execute the preliminary processing forthe image data output from the pixels present in the corresponding block111 a.

Since this makes it possible to engage the plurality of selection units322 in concurrent execution of preliminary processing for the imagedata, the processing onus at each selection unit 322 can be reduced, andan optimal image can be generated in a short period of time by usingsets of image data individually generated in areas with differentimage-capturing conditions applied thereto.

It is to be noted that when describing the relationship between a givenblock 111 a and the pixels present in the particular block 111 a, theblock 111 a may be referred to as the block 111 a to which the pixelsbelong in the following explanation. In addition, a block 111 a mayotherwise be referred to as a unit block and a plurality of blocks 111 agrouped together, i.e., an aggregate of unit blocks may be referred toas a combined block.

FIG. 21 presents a sectional view of a stacked image sensor 100A. Thestacked image sensor 100A includes the image processing chip 114, whichexecutes the preliminary processing described above and imageprocessing, in addition to a back-side illumination-type image-capturingchip 111, a signal processing chip 112 and a memory chip 113. Namely,the image processing unit 32 c is disposed at the image processing chip114.

The image-capturing chip 111, the signal processing chip 112, the memorychip 113 and the image processing chip 114 are stacked one upon anotherand are electrically connected with one another via bumps 109constituted of a material such as Cu having electric conductivity.

A plurality of bumps 109 are disposed at the surface of the memory chip113 and the surface of the image processing chip 114 that face oppositeeach other. As these bumps 109 are aligned relative to each other andthe memory chip 113 and the image processing chip 114 are pressedagainst each other, the bumps 109, having been aligned with each other,become bonded, thereby achieving electrical connection.

<Data Selection Processing>

In the second embodiment, once the setting unit 34 b has divided theimaging field into a plurality of areas, image-capturing conditions canbe set (adjusted), for an area selected by the user or for an areadetermined by the control unit 34, as in the first embodiment. Ifdifferent image-capturing conditions have been set for individual areas,the control unit 34 initiates the following data selection processing asneeded.

1. When Executing Image Processing

1-1. When uniform image-capturing conditions are selected for the targetpixel P and a plurality of reference pixels around the target pixel P

In this case, the corresponding selection unit 322 selects all the imagedata obtained at the plurality of reference pixels and outputs theselected image data to the generation unit 323. The generation unit 323executes image processing by using the image data from the plurality ofreference pixels.

1-2. When the image-capturing conditions set for the target pixel P aredifferent from the image-capturing conditions set for at least one ofthe plurality of reference pixels around the target pixel P

The image-capturing conditions applied at the target pixel P are firstimage-capturing conditions, the image-capturing conditions applied atsome of the plurality of reference pixels are also the firstimage-capturing conditions and the image-capturing conditions applied atthe rest of the reference pixels are second image-capturing conditions.

In this situation, a selection unit 322 corresponding to a block 111 ato which a reference pixel with the first image-capturing conditionsapplied thereto belongs, and a selection unit 322 corresponding to ablock 111 a to which a reference pixel with the second image-capturingconditions applied thereto belongs, execute data selection processing,as explained in (example 1) through (example 3) below with regard to theimage data at the reference pixels. The generation unit 323 thenexecutes image processing so as to generate through calculation imagedata for the target pixel P by referencing the image data at thereference pixels selected through the data selection processing.

Example 1

For instance, the first image-capturing conditions and the secondimage-capturing conditions may differ from each other only in the ISOsensitivity, with an ISO sensitivity setting of 100 selected for thefirst image-capturing conditions and an ISO sensitivity setting of 800selected for the second image-capturing conditions. Under thesecircumstances, the selection unit 322 corresponding to a block 111 a towhich a reference pixel with the first image-capturing conditionsapplied thereto belongs, selects the image data obtained under the firstimage-capturing conditions. However, the selection unit 322corresponding to a block 111 a to which a reference pixel with thesecond image-capturing conditions applied thereto belongs, does notselect the image data obtained under the second image-capturingconditions. In other words, the image data obtained under the secondimage-capturing conditions, which are different from the firstimage-capturing conditions, are not used in the image processing.

Example 2

The first image-capturing conditions and the second image-capturingconditions may differ from each other only in the shutter speed, with ashutter speed setting of 1/1000 sec selected for the firstimage-capturing conditions and a shutter speed setting of 1/100 secselected for the second image-capturing conditions. Under thesecircumstances, the selection unit 322 corresponding to a block 111 a towhich a reference pixel with the first image-capturing conditionsapplied thereto belongs, selects the image data obtained under the firstimage-capturing conditions. However, the selection unit 322corresponding to a block 111 a to which a reference pixel with thesecond image-capturing conditions applied thereto belongs, does notselect the image data obtained under the second image-capturingconditions. In other words, the image data obtained under the secondimage-capturing conditions, which are different from the firstimage-capturing conditions, are not used in the image processing.

Example 3

The first image-capturing conditions and the second image-capturingconditions may differ from each other only in the frame rate (withuniform electric charge accumulation time) with a frame rate setting of30 fps selected for the first image-capturing conditions and a framerate setting of 60 fps selected for the second image-capturingconditions. Under these circumstances, the selection unit 322corresponding to a block 111 a to which a reference pixel with the firstimage-capturing conditions applied thereto belongs, selects the imagedata obtained under the first image-capturing conditions. In addition,the selection unit 322 corresponding to a block 111 a to which areference pixel with the second image-capturing conditions appliedthereto belongs, selects image data in frame images obtained with timingclose to the timing with which frame images have been obtained under thefirst image-capturing conditions (30 fps) among the sets of image dataobtained under the second image-capturing conditions (60 fps) at thereference pixel. Namely, image data expressing frame images obtainedwith timing close to the timing with which frame images have beenobtained under the first image-capturing conditions (30 fps) are used inthe image processing, but image data obtained with timing different fromthe timing with which the frame images have been obtained under thefirst image-capturing conditions (30 fps) among the sets of image dataat the reference pixel, are not used in the image processing.

It is to be noted that data selection processing will be executed inmuch the same way when the image-capturing conditions applied at thetarget pixel P are the second image-capturing conditions and theimage-capturing conditions applied at reference pixels around the targetpixel P are the first image-capturing conditions. Namely, a selectionunit 322 corresponding to a block 111 a to which a reference pixel withthe first image-capturing conditions applied thereto belongs, and aselection unit 322 corresponding to a block 111 a to which a referencepixel with the second image-capturing conditions applied theretobelongs, execute data selection processing, as has been explained inreference (example 1) to (example 3) for the image data at the referencepixels.

It is to be noted that as has been explained earlier, image-capturingconditions indicating a slight difference are regarded as uniformimage-capturing conditions.

The generation unit 323 executes image processing such as defectivepixel correction processing, color interpolation processing, edgeenhancement processing and noise reduction processing based upon theimage data at the reference pixels selected by the selection units 322,as does the image processing unit 33 (generation unit 33 c) in the firstembodiment.

FIG. 22 is a schematic diagram illustrating how image data (hereafterreferred to as first image data) from individual pixels present in anarea (hereafter referred to as a first area 141) of the image-capturingsurface, with the first image-capturing conditions applied thereto, andimage data (hereafter referred to as second image data) from individualpixels present in an area (hereafter referred to as a second area 142)at the image-capturing surface, with the second image-capturingconditions applied thereto, are processed. It is to be noted that theprocessing in FIG. 22 is executed for the target pixel P with the firstimage-capturing conditions applied thereto in (example 1) and (example2) described earlier.

The first image data expressing an image captured under the firstimage-capturing conditions are output from each pixel present in thefirst area 141, whereas the second image data expressing an imagecaptured under the second image-capturing conditions are output fromeach pixel present in the second area 142. The first image data areoutput to the selection unit 322 that corresponds to the block 111 a towhich the particular pixel having generated the first image databelongs, among the plurality of selection units 322 disposed at theprocessing chip 114. In the following description, a plurality ofselection units 322, each corresponding to one of a plurality of blocks111 a to which pixels having generated individual sets of first imagedata belong, will be referred to as a first selection unit 151.

Likewise, the second image data are output to the selection unit 322that corresponds to the block 111 a, to which the particular pixelhaving generated the second image data belongs, among the plurality ofselection units 322 disposed at the processing chip 114. In thefollowing description, a plurality of selection units 322, eachcorresponding to one of a plurality of blocks 111 a, to which the pixelshaving generated individual sets of second image data belong, will bereferred to as a second selection unit 152.

For instance, the target pixel P may be present in the first area 141,and in this case, the first selection unit 151 selects the image data atthe target pixel P and image data obtained at reference pixels bycapturing an image under the first image-capturing conditions andoutputs the selected image data to the generation unit 323. While imagedata in a single block are selected via the selection unit 322 in thisexample, image data from another block, generated by capturing an imageunder the first image-capturing conditions, may be selected as well. Inthe latter case, the selection unit 322, to which the image data fromthe target pixel P have been input and the selection unit 322corresponding to another block with the first image-capturing conditionsapplied thereto for image-capturing operation simply need to exchangethe information 181 pertaining to the first image-capturing conditionsrequired for data selection processing. The second selection unit 152does not select image data at reference pixels generated by capturing animage under the second image-capturing conditions and thus does notoutput the image data at the reference pixels with the secondimage-capturing conditions applied thereto to the generation unit 323.It is to be noted that the second selection unit 152 receives theinformation 181 pertaining to the first image-capturing conditionsrequired in the data selection processing from, for instance, the firstselection unit 151.

Likewise, if the target pixel P is present in the second area, and inthis case, the second selection unit 152 selects the image data at thetarget pixel P and image data obtained at reference pixels by capturingan image under the second image-capturing conditions and outputs theselected image data to the generation unit 323. The first selection unit151 does not select the image data at reference pixels generated bycapturing an image under the first image-capturing conditions, and thusdoes not output image data at the reference pixels with the firstimage-capturing conditions applied thereto to the generation unit 323.It is to be noted that the first selection unit 151 receives informationpertaining to the second image-capturing conditions required in the dataselection processing from, for instance, the second selection unit 152.

Once the preliminary processing has been executed as described above,the generation unit 323 executes image processing such as defectivepixel correction processing, color interpolation processing, edgeenhancement processing and noise reduction processing based upon theimage data provided via the first selection unit 151 and the secondselection unit 152, and outputs image data resulting from the imageprocessing.

2. When Executing Focus Detection Processing

As in the first embodiment, the control unit 34 (AF operation unit 34 d)executes focus detection processing by using image data corresponding toa specific position (focus detection position) on the imaging field. Itis to be noted that if different image-capturing conditions have beenset in different areas and the focus detection position set for an AFoperation is located at a boundary between areas, i.e., if the focusdetection position is split over the first area and the second area, thecontrol unit 34 (AF operation unit 34 d) in the embodiment engages theselection units 322 in data selection processing as explained in 2-2below.

2-1. When the image data from the pixels present within the frame 170 inFIG. 13 do not include both focus detection signal data obtained underthe first image-capturing conditions and focus detection signal dataobtained under the second image-capturing conditions

In this case, the selection units 322 select the focus detection signaldata from all the pixels present in the frame 170 and output theselected data to the generation unit 323. The control unit 34 (AFoperation unit 34 d) executes focus detection processing by using thefocus detection signal data obtained at the focus detection pixelswithin the frame 170.

2-2. When the image data from the pixels present within the frame 170 inFIG. 13 include both focus detection signal data obtained under thefirst image-capturing conditions and focus detection signal dataobtained under the second image-capturing conditions output

The control unit 34 (AF operation unit 34 d) engages the selection units322 that correspond to the blocks 111 a to which the pixels in the frame170 belong, in data selection processing, as described in (example 1)through (example 3) below. The control unit 34 (AF operation unit 34 d)then executes focus detection processing by using the focus detectionsignal data selected through the data selection processing.

Example 1

For instance, the first image-capturing conditions and the secondimage-capturing conditions may differ from each other only in the ISOsensitivity, with an ISO sensitivity setting of 100 selected for thefirst image-capturing conditions and ISO sensitivity setting of 800selected for the second image-capturing conditions. Under thesecircumstances, a selection unit 322 corresponding to a block 111 a, towhich a pixel with the first image-capturing conditions applied theretobelongs, selects the focus detection signal data obtained under thefirst image-capturing conditions. However, a selection unit 322corresponding to a block 111 a to which a pixel with the secondimage-capturing conditions applied thereto belongs does not select thefocus detection signal data obtained under the second image-capturingconditions. In other words, only the focus detection signal dataobtained under the first image-capturing conditions, among the sets offocus detection signal data obtained at pixels present inside the frame170, are used for the focus detection processing without using the imagedata obtained under the second image-capturing conditions, differentfrom the first image-capturing conditions, for the focus detectionprocessing.

Example 2

The first image-capturing conditions and the second image-capturingconditions may differ from each other only in the shutter speed, with ashutter speed setting of 1/1000 sec selected for the firstimage-capturing conditions and a shutter speed setting of 1/100 secselected for the second image-capturing conditions. Under thesecircumstances, a selection unit 322 corresponding to a block 111 a towhich a pixel with the first image-capturing conditions applied theretobelongs, selects the focus detection signal data obtained under thefirst image-capturing conditions. However, a selection unit 322corresponding to a block 111 a to which a pixel with the secondimage-capturing conditions applied thereto belongs does not select thefocus detection signal data obtained under the second image-capturingconditions. In other words, only the focus detection signal dataobtained under the first image-capturing conditions, among the sets offocus detection signal data obtained at pixels present inside the frame170, are used for the focus detection processing without using the focusdetection signal data obtained under the second image-capturingconditions, different from the first image-capturing conditions, for thefocus detection processing.

Example 3

The first image-capturing conditions and the second image-capturingconditions may differ from each other only in the frame rate (withuniform electric charge accumulation time) with a frame rate setting of30 fps selected for the first image-capturing conditions and a framerate setting of 60 fps selected for the second image-capturingconditions. Under these circumstances, a selection unit 322corresponding to a block 111 a to which a pixel with the firstimage-capturing conditions applied thereto belongs, selects the focusdetection signal data obtained under the first image-capturingconditions. In addition, a selection unit 322 corresponding to a block111 a to which a pixel with the second image-capturing conditionsapplied thereto belongs, selects focus detection signal data in frameimages obtained under the second image-capturing conditions (60 fps)with timing close to the timing with which a frame image have beenobtained under the first image-capturing conditions (30 fps). Namely,focus detection signal data expressing frame images obtained with timingclose to the timing with which the focus detection signal data for frameimages have been obtained under the first image-capturing conditions,among sets of focus detection signal data obtained under the secondimage-capturing conditions (60 fps) are used in the focus detectionprocessing, but focus detection signal data obtained with timingsignificantly different from the timing with which the frame images havebeen obtained under the first image-capturing conditions (30 fps) arenot used in the focus detection processing.

It is to be noted that as has been explained earlier, image-capturingconditions indicating a slight difference are regarded as uniformimage-capturing conditions.

In addition, while the focus detection signal data obtained under thefirst image-capturing conditions, among the sets of focus detectionsignal data obtained in the area enclosed with the frame 170, areselected in (example 1) and (example 2) described above, the focusdetection signal data obtained under the second image-capturingconditions, among the sets of focus detection signal data in the areawithin the frame 170, may be selected instead.

It is to be also noted that if the focus detection position is splitover the first area and the second area with the first area assuming agreater areal size than the second area, it is desirable to select theimage data obtained under the first image-capturing conditions, whereasif the second area takes a greater areal size than the first area, it isdesirable to select image data obtained under the second image-capturingconditions.

FIG. 23 presents a schematic diagram illustrating how first signal dataand second signal data are processed in relation to the focus detectionprocessing. It is to be noted that FIG. 23 illustrates how processing isexecuted with the focus detection signal data obtained under the firstimage-capturing conditions, selected from the sets of signal dataobtained in the area enclosed by the frame 170, and with the focusdetection signal data obtained under the second image-capturingconditions also selected from the sets of signal data obtained in thearea enclosed by the frame 170.

First signal data to be used for focus detection, generated by capturingan image under the first image-capturing conditions, are output fromeach of the pixels present in the first area 141, whereas second signaldata to be used for focus detection, generated by capturing an imageunder the second image-capturing conditions, are output from each of thepixels present in the second area 142. The first signal data originatingfrom the first area 141 are output to the first selection unit 151.Likewise, the second signal data originating from the second area 142are output to the second selection unit 152.

The first processing unit 151 selects the first signal data obtainedunder the first image-capturing conditions and outputs the selected datato the AF operation unit 34 d. The second processing unit 152 selectsthe second signal data obtained under the second image-capturingconditions and outputs the selected data to the AF operation unit 34 d.

Once the preliminary processing has been executed as described above,the AF operation unit 34 d calculates a first defocus quantity basedupon the first signal data provided via the first processing unit 151.In addition, the AF operation unit 34 d calculates a second defocusquantity based upon the second signal data provided from the firstprocessing unit 151. Then, the AF operation unit 34 d outputs a drivesignal generated based upon the first defocus quantity and the seconddefocus quantity to be used to drive the focus lens in theimage-capturing optical system 31 to the in-focus position.

It is to be noted that if the focus detection signal data obtained underthe first image-capturing conditions, among the sets of signal dataobtained in the area within the frame 170, are selected but the focusdetection signal data obtained under the second image-capturingconditions located in the area within the frame 170 are not selected, asin (example 1) and (example 2) described above, the following processingis executed with regard to the first signal data and the second signaldata.

When focus detection processing is executed by using the first signaldata obtained under the first image-capturing conditions, for instance,the first processing unit 151 selects the first signal data obtainedunder the first image-capturing conditions and outputs the selectedfirst signal data to the generation unit 323. The second processing unit152, however, does not select the second signal data obtained under thesecond image-capturing conditions and does not output the second signaldata at the reference pixels with the second image-capturing conditionsapplied thereto to the generation unit 323. It is to be noted that thesecond processing unit 152 receives the information 181 pertaining tothe first image-capturing conditions required in the data selectionprocessing from, for instance, the first processing unit 151.

Once the preliminary processing has been executed as described above,the AF operation unit 34 d executes focus detection processing basedupon the first signal data provided via the first processing unit 151and outputs a drive signal, generated based upon the arithmeticoperation results, to be used to drive the focus lens in theimage-capturing optical system 31, to the in-focus position.

An explanation will be given next on an example in which a focusingtarget subject, onto which focus is to be adjusted, takes a positionastride an area with the first image-capturing conditions appliedthereto and an area with the second image-capturing conditions appliedthereto. When the focusing target subject takes a position astride anarea with the first image-capturing conditions applied thereto and anarea with the second image-capturing conditions applied thereto, a,selection unit 322 corresponding to a block 111 a to which a pixel withthe first image-capturing conditions applied thereto belongs, selectsthe first signal data generated under the first image-capturingconditions, to be used for focus detection. In addition, a selectionunit 322 corresponding to a block 111 a, to which a pixel with thesecond image-capturing conditions applied thereto belongs, selects thesecond signal data generated under the second image-capturingconditions, to be used for focus detection. The control unit 34 (AFoperation unit 34 d) then calculates a first defocus quantity based uponthe first focus detection signal data having been selected. The controlunit 34 (AF operation unit 34 d) further calculates a second defocusquantity based upon the second focus detection signal data having beenselected. The control unit 34 (AF operation unit 34 d) next executesfocus detection processing based upon the first defocus quantity and thesecond defocus quantity. In more specific terms, the control unit 34 (AFoperation unit 34 d) calculates a lens drive distance by, for instance,calculating the average of the first defocus quantity and the seconddefocus quantity. As an alternative, the control unit 34 (AF operationunit 34 d) may select a value representing either the first defocusquantity or the second defocus quantity, that indicates a smaller drivedistance for the lens. As a further alternative, the control unit 34 (AFoperation unit 34 d) may select a value representing either the firstdefocus quantity or the second defocus quantity that indicates that thesubject is located closer to the camera.

In addition, when the focusing target subject is present astride an areawith the first image-capturing conditions applied thereto and an areawith the second image-capturing conditions applied thereto, the controlunit 34 (AF operation unit 34 d) may select the focus detectionphotoelectric conversion signals generated in the area occupied by thegreater part of the subject. For instance, 70% of the facial area, i.e.,the focusing target subject, may be present in the area with the firstimage-capturing conditions applied thereto with the remaining 30% of thefacial area in the second area, and in such case, the control unit 34(AF operation unit 34 d) will select the focus detection photoelectricconversion signals generated under the first image-capturing conditions.It is to be noted that the areal ratio (%) described above simplyrepresents an example, in the present invention is not limited to thisexample.

3. When Executing Subject Detection Processing

The processing executed when different image-capturing conditions areset for different areas and a search range 190 includes a boundarybetween areas will be explained next.

3-1. When the image data available within the search range 190 in FIG.14 do not include both image data obtained under the firstimage-capturing conditions and image data obtained under the secondimage-capturing conditions

In this case, the selection units 322 select the image data from all thepixels present in the search range 190 and output the selected imagedata to the generation unit 323. The control unit 34 (AF operation unit34 d) executes subject detection processing by using the image dataobtained at the pixels within the search range 190.

3-2. When the image data within the search range 190 in FIG. 14 includeboth image data obtained under the first image-capturing conditions andimage data obtained under the second image-capturing conditions

(a) When the first image-capturing conditions and the secondimage-capturing conditions differ from each other only in the ISOsensitivity or in the shutter speed, as in (example 1) or (example 2)described earlier in reference to the focus detection processing

Under these circumstances, the control unit 34 (object detection unit 34a) engages each selection unit 322 (first selection unit 151)corresponding to a block 111 a, to which a pixel with the firstimage-capturing conditions applied thereto belongs so as to select theimage data obtained under the first image-capturing conditions to beused in the subject detection processing, among the sets of image dataavailable in the search range 190, as illustrated in FIG. 24. It is tobe noted that FIG. 24 presents a schematic diagram illustrating howfirst image data and second image data are processed in relation to thesubject detection processing.

The control unit 34 (object detection unit 34 a) then executes thesubject detection processing by using the image data selected throughthe data selection processing.

The control unit 34 (object detection unit 34 a) also engages eachselection unit 322 (second selection unit 152) corresponding to a block111 a, to which a pixel with the second image-capturing conditionsapplied thereto belongs so as to select the image data obtained underthe second image-capturing conditions to be used in the subjectdetection processing, among the sets of image data available in thesearch range 190. The control unit 34 (object detection unit 34 a) thenexecutes the subject detection processing by using the image dataselected through the data selection processing. The control unit 34(object detection unit 34 a) is able to detect a subject present withinthe search range 190 by combining the subject area detected based uponthe image data obtained under the first image-capturing conditions andthe subject area detected based upon the image data obtained under thesecond image-capturing conditions at their boundary.

(b) When the first image-capturing conditions and the secondimage-capturing conditions are different from each other only in theframe rate, as in (example 3) explained earlier in reference to thefocus detection processing

In this case, the control unit 34 (object detection unit 34 a) engageseach selection unit 322 (first selection unit 151) corresponding to ablock 111 a, to which a pixel with the first image-capturing conditionsapplied thereto belongs, so as to select the image data obtained underthe first image-capturing conditions to be used in the subject detectionprocessing, among the sets of image data available in the search range190. The control unit 34 (object detection unit 34 a) then executes thesubject detection processing by using the image data selected throughthe data selection processing. In addition, the control unit 34 (objectdetection unit 34 a) engages each selection unit 322 (second selectionunit 152) corresponding to a block 111 a, to which a pixel with thesecond image-capturing conditions applied thereto belongs, so as toselect only image data expressing frame images that have been obtainedwith timing close to the timing with which frame images have beenobtained under the first image-capturing conditions (30 fps), among thesets of image data available in the search range 190, as the image datacorresponding to the second image-capturing conditions (60 fps) to beused in the subject detection processing. The control unit 34 (objectdetection unit 34 a) then executes the subject detection processing byusing the image data having been selected through the data selectionprocessing. The control unit 34 (object detection unit 34 a) is able todetect a subject present within the search range 190 by combining thesubject area detected based upon the image data obtained under the firstimage-capturing conditions and the subject area detected based upon theimage data obtained under the second image-capturing conditions at theirboundary.

It is to be noted that if the search range 190 is split over the firstarea and the second area with the first area assuming a greater arealsize than the second area, only the image data obtained under the firstimage-capturing conditions may be selected without selecting any imagedata obtained under the second image-capturing conditions. If, on theother hand, the second area assumes a greater areal size than the firstarea, only the image data obtained under the second image-capturingconditions may be selected without selecting any image data obtainedunder the first image-capturing conditions.

4. When Setting Image-Capturing Conditions

Data selection processing executed when exposure conditions aredetermined by re-executing photometering operation with the imagingfield divided into a plurality of areas with different image-capturingconditions set for different areas will be explained.

4-1. When the image data available within a photometering range do notinclude both image data obtained under the first image-capturingconditions and image data obtained under the second image-capturingconditions

In this case, the selection units 322 select the image data from all thepixels present in the photometering range and output the selected datato the generation unit 323. The control unit 34 (setting unit 34 b)executes exposure calculation processing by using the image dataobtained at the pixels within the photometering range.

4-2. When the image data available within the photometering rangeinclude both image data obtained under the first image-capturingconditions and image data obtained under the second image-capturingconditions

(a) When the first image-capturing conditions and the secondimage-capturing conditions differ from each other only in the ISOsensitivity or in the shutter speed, as in (example 1) or (example 2)described earlier in reference to the focus detection processing

Under these circumstances, the control unit 34 (object detection unit 34a) engages each selection unit 322 (first selection unit 151)corresponding to a block 111 a, to which a pixel with the firstimage-capturing conditions applied thereto belongs, so as to select theimage data obtained under the first image-capturing conditions to beused in the subject detection processing, among the sets of image dataavailable in the photometering range, as illustrated in FIG. 25 in amanner similar to that described in (a) pertaining to the subjectdetection processing. The control unit 34 (object detection unit 34 a)also engages each selection unit 322 corresponding to a block 111 a towhich a pixel with the second image-capturing conditions applied theretobelongs, so as to select the image data obtained under the secondimage-capturing conditions to be used in the exposure calculationprocessing, among the sets of image data available in the photometeringrange. It is to be noted that FIG. 25 presents a schematic diagramillustrating how the first image data and the second image data areprocessed when setting image-capturing conditions through exposurecalculation processing or the like.

The control unit 34 (setting unit 34 b) then executes exposurecalculation processing individually for the area with the firstimage-capturing conditions applied thereto and the area with the secondimage-capturing conditions applied thereto by using the image dataselected through the data selection processing. Namely, the control unit34 (setting unit 34 b) executes data selection processing so as tophotometer the individual areas and executes exposure calculationprocessing by using the image data selected through the data selectionprocessing if a plurality of areas with different image-capturingconditions applied thereto are present within the photometering range.

(b) When the first image-capturing conditions and the secondimage-capturing conditions are different from each other only in theframe rate, as in (example 3) explained earlier in reference to thefocus detection processing

In this case, the control unit 34 (object detection unit 34 a) engageseach selection unit 322 corresponding to a block 111 a, to which a pixelwith the first image-capturing conditions applied thereto belongs, so asto select the image data obtained under the first image-capturingconditions to be used in the exposure calculation processing, among thesets of image data available in the photometering range, as has beendescribed in (b) in relation to the subject detection processing. Inaddition, the control unit 34 (object detection unit 34 a) engages eachselection unit 322 corresponding to a block 111 a to which a pixel withthe second image-capturing conditions applied thereto belongs, so as toselect only image data expressing frame images that have been obtainedwith timing close to the timing with which frame images have beenobtained under the first image-capturing conditions (30 fps), among thesets of image data available in the photometering range, as the imagedata corresponding to the second image-capturing conditions (60 fps) tobe used in the exposure calculation processing, as in (example 3)explained in relation to the focus detection processing.

The control unit 34 (setting unit 34 b) then executes the exposurecalculation processing by using the image data selected through the dataselection processing as in (a) above.

It is to be noted that if the photometering range is split over thefirst area and the second area with the first area assuming a greaterareal size than the second area, only the image data obtained under thefirst image-capturing conditions may be selected without selecting anyimage data obtained under the second image-capturing conditions. If, onthe other hand, the second area assumes a greater areal size than thefirst area, only the image data obtained under the secondimage-capturing conditions may be selected without selecting any imagedata obtained under the first image-capturing conditions.

The following advantages and operations are achieved through the thirdembodiment described above.

(1) The camera 1C includes an image sensor 32 a capable of capturingimages under image-capturing conditions set from one unit block toanother defined at the image-capturing surface thereof, which generatesfirst image data from a first area made up with at least one unit blockby capturing an image under first image-capturing conditions, and secondimage data from a second area made up with at least one unit block bycapturing an image under second image-capturing conditions differentfrom the first image-capturing conditions. The camera 1C includes aplurality of selection units 322, each disposed in correspondence to oneof the unit blocks or in correspondence to a combined block made up witha plurality of unit blocks. The plurality of selection units 322 eacheither select or do not select the image data from the correspondingunit block or from the unit blocks making up the corresponding combinedblock. The image sensor 32 a is disposed at a back-side illuminationimage-capturing chip 111. The plurality of selection units 322 aredisposed at an image-capturing chip 114.

These structural features make it possible to concurrently execute dataselection processing to select image data at the plurality of selectionunits 322 and, as a result, the processing onus on the individualselection units 322 is reduced.

(2) The back-side illumination image-capturing chip 111 and the imageprocessing chip 114 are stacked one upon the other. This means that theimage sensor 32 a and the image processing unit 32 c can achieveconnection with ease.

(3) The camera 1C includes a generation unit 323 that generates an imageby using the selected image data, having been selected via the selectionunits 322. Since the preliminary processing can be executed by theplurality of selection units 322 through concurrent processing, thelength of time required for image generation can be reduced.

(4) The camera 1C includes an image sensor 32 a capable of capturingimages under different image-capturing conditions set from one unitblock to another defined at the image-capturing surface thereof, whichgenerates first image data from a first area made up with at least oneunit block by capturing an optical image formed with light havingentered therein via an image-capturing optical system, under firstimage-capturing conditions, and second image data from a second areamade up with at least one unit block by capturing an optical imageformed with light having entered therein under second image-capturingconditions different from the first image-capturing conditions. Thecamera 1C includes a plurality of selection units 322, each disposed incorrespondence to one of the unit blocks or in correspondence to acombined block made up with a plurality of unit blocks. The plurality ofselection units 322 each either select or do not select the image datafrom the corresponding unit block or from the unit blocks making up thecorresponding combined block. The camera 1C also includes an AFoperation unit 34 d that detects information used to drive theimage-capturing optical system based upon the image data having beenselected. The image sensor 32 a is disposed at a back-side illuminationimage-capturing chip 111. The plurality of correction units 322 aredisposed at an image-capturing chip 114.

These structural features make it possible to concurrently execute dataselection processing to select image data at the plurality of selectionunits 322 and, as a result, the processing onus on the individualselection units 322 is reduced. Furthermore, since the preliminaryprocessing can be executed quickly by the plurality of selection units322 through parallel processing, the length of time to elapse before theAF operation unit 34 d can start executing the focus detectionprocessing is reduced, thereby making it possible to speed up the focusdetection processing.

(5) The camera 1C includes an image sensor 32 a capable of capturingimages under different image-capturing conditions set from one unitblock to another defined at the image-capturing surface thereof, whichgenerates first image data from a first area made up with at least oneunit block by capturing a subject image formed with light having enteredtherein via an image-capturing optical system, under firstimage-capturing conditions, and second image data from a second areamade up with at least one unit block by capturing a subject image formedwith light having entered therein under second image-capturingconditions different from the first image-capturing conditions. Thecamera 1C includes a plurality of selection units 322, each disposed incorrespondence to one of the unit blocks or in correspondence to acombined block made up with a plurality of unit blocks. The plurality ofselection units 322 each either select or do not select the image datafrom the corresponding unit block or from the unit blocks making up thecorresponding combined block. The camera 1C also includes an objectdetection unit 34 a that detects a target object in the subject imagebased upon the selected image data having been selected. The imagesensor 32 a is disposed at a back-side illumination image-capturing chip111. The plurality of correction units 322 are disposed at animage-capturing chip 114.

These structural features make it possible to concurrently execute dataselection processing to select image data at the plurality of selectionunits 322 and, as a result, the processing onus on the individualselection units 322 is reduced. Furthermore, since the preliminaryprocessing can be executed quickly by the plurality of selection units322 through parallel processing, the length of time to elapse before theobject detection unit 34 a can start executing subject detectionprocessing is reduced, thereby making it possible to speed up thesubject detection processing.

(6) The camera 1C includes an image sensor 32 a capable of capturingimages under different image-capturing conditions set from one unitblock to another defined at the image-capturing surface thereof, whichgenerates first image data from a first area made up with at least oneunit block by capturing an optical image, formed with light havingentered therein via an image-capturing optical system, under firstimage-capturing conditions, and second image data from a second areamade up with at least one unit block by capturing an image formed withlight having entered therein under second image-capturing conditionsdifferent from the first image-capturing conditions. The camera 1Cincludes a plurality of selection units 322, each disposed incorrespondence to one of the unit blocks or in correspondence to acombined block made up with a plurality of unit blocks. The plurality ofselection units 322 each either select or do not select the image datafrom the corresponding unit block or from the unit blocks making up thecorresponding combined block. The camera 1C also includes a setting unit34 b that sets photographing conditions based upon the selected imagedata having been selected. The image sensor 32 a is disposed at aback-side illumination image-capturing chip 111. The plurality ofcorrection units 322 are disposed at an image-capturing chip 114.

These structural features make it possible to concurrently execute dataselection processing to select image data at the plurality of selectionunits 322 and, as a result, the processing onus on the individualselection units 322 is reduced. Furthermore, since the preliminaryprocessing can be executed quickly by the plurality of selection units322 through parallel processing, the length of time to elapse before theAF setting unit 34 b can start executing image-capturing conditionsetting processing is reduced, thereby making it possible to speed upthe image-capturing condition setting processing.

Variations of the Third Embodiment

The following variations are also within the scope of the presentinvention, and one of the variations or a plurality of variations may beadopted in combination with the embodiment described above.

(Variation 10) An explanation will be given on how first image data andsecond image data are processed with the first area and the second areaset at the image-capturing surface of the image sensor 32 a, as shown inFIG. 16(a) through FIG. 16(c) in reference to which Variation 1 of thefirst and second embodiments have been described.

In this variation, too, pixel signals read out from the image sensor 32a having captured an image for one frame are used to generate a firstimage based upon image signals read out from the first area and a secondimage based upon image signals read out from the second area, as inVariation 1, in each of the examples presented in FIG. 16(a) throughFIG. 16(c). In this variation, the control unit 34 uses the first imageas a display image and uses the second image for purposes of detection,as in Variation 1.

It is assumed that first image-capturing conditions are set for thefirst area where the first image is captured and that secondimage-capturing conditions different from the first image-capturingconditions are set for the second area where the second image iscaptured.

1. An example in which the first image-capturing conditions are uniformover the entire first area at the image-capturing surface and the secondimage-capturing conditions, too, are uniform over the entire second areaat the image-capturing surface, will be explained in reference to FIG.26. FIG. 26 presents a schematic diagram illustrating how first imagedata and second image data are processed.

First image data obtained by capturing an image under the firstimage-capturing conditions are output from each of the pixels present inthe first area 141, whereas second image data and second signal dataobtained by capturing an image under the second image-capturingconditions are output from each of the pixels present in the second area142. The first image data from the first area 141 are output to thefirst selection unit 151. Likewise, the second image data and the secondsignal data from the second area 142 are output to the second selectionunit 152.

In this example, uniform first image-capturing conditions are set forthe entire first area at the image-capturing surface and accordingly,the first selection unit 151 selects the first image data from all thepixels present in the first area. In addition, since uniform secondimage-capturing conditions are set for the entire second area at theimage-capturing surface, the second selection unit 152 selects thesecond image data from all the pixels present in the second area. It isto be noted that since the first image-capturing conditions and thesecond image-capturing conditions are different from each other, thesecond selection unit 152 does not select the second image data as datato be used in the image processing executed for the image data in thefirst area.

In addition, the second selection unit 152 receives the information 181pertaining to the first image-capturing conditions from, for instance,the first selection unit 151.

The generation unit 323 executes image processing such as defectivepixel correction processing, color interpolation processing, edgeenhancement processing and noise reduction processing based upon thefirst image data provided via the first selection unit 151 and outputsimage data having undergone the image processing.

The object detection unit 34 a executes processing so as to detectsubject elements based upon the second image data provided via thesecond selection unit 152 and outputs the detection results.

The setting unit 34 b executes image-capturing condition calculationprocessing such as exposure calculation processing based upon the secondimage data provided via the second selection unit 152. Then, based uponthe calculation results, it divides the image-capturing surface at theimage-capturing unit 32 into a plurality of areas containing detectedsubject elements and resets image-capturing conditions for the pluralityof areas.

The AF operation unit 34 d executes focus detection processing basedupon the second signal data provided via the second selection unit 152and outputs a drive signal, generated based upon the arithmeticoperation results, to be used to drive the focus lens in theimage-capturing optical system 31 to the in-focus position.

2. Another example in which the first image-capturing conditions arealtered depending upon the location at the image-capturing surface,i.e., different first image-capturing conditions are set from one arealportion to another within the first area, and uniform secondimage-capturing conditions are set for the entire second area at theimage-capturing surface, will be explained in reference to FIG. 27. FIG.27 presents a schematic diagram illustrating how first image data,second image data and second signal data are processed.

First image data obtained by capturing an image under the firstimage-capturing conditions are output from each of the pixels present inthe first area 141, whereas second image data or second signal dataobtained by capturing an image under the second image-capturingconditions are output from each of the pixels present in the second area142. The first image data from the first area 141 are output to thefirst selection unit 151. Likewise, the second image data from thesecond area are output to the second selection unit 152.

As described earlier, the first image-capturing conditions are altereddepending upon the location at the image-capturing surface in thisexample. In other words, different first image-capturing conditions areset for different areal portions within the first area. The firstselection unit 151 selects only first image data obtained under specificimage-capturing conditions, among the sets of first image data obtainedat the individual pixels present in the first area but does not selectfirst image data obtained under other image-capturing conditions. Inaddition, since uniform second image-capturing conditions are set forthe entire second area at the image-capturing surface, the secondselection unit 152 selects the second image data at all the pixelspresent in the second area. It is to be noted that since the firstimage-capturing conditions and the second image-capturing conditions aredifferent from each other, the second selection unit 152 does not selectthe second image data as data to be used in the image processingexecuted for the image data in the first area.

In addition, the second selection unit 152 receives the information 181pertaining to the first image-capturing conditions from, for instance,the first selection unit 151.

The generation unit 323 executes image processing such as defectivepixel correction processing, color interpolation processing, edgeenhancement processing and noise reduction processing based upon thepart of the first image data having been selected via the firstselection unit 151 and outputs image data having undergone the imageprocessing.

The object detection unit 34 a executes processing so as to detectsubject elements based upon the second image data provided via thesecond selection unit 152 and outputs the detection results.

The setting unit 34 b executes image-capturing condition calculationprocessing such as exposure calculation processing based upon the secondimage data provided via the second selection unit 152. Then, based uponthe calculation results, it divides the image-capturing surface at theimage-capturing unit 32 into a plurality of areas containing detectedsubject elements and resets image-capturing conditions for the pluralityof areas.

The AF operation unit 34 d executes focus detection processing basedupon the second signal data, generated based upon the arithmeticoperation results provided via the second selection unit 152 and outputsa drive signal to be used to drive the focus lens in the image-capturingoptical system 31 to the in-focus position.

3. Yet another example in which uniform first image-capturing conditionsare set for the first area at the image-capturing surface, but thesecond image-capturing conditions are altered depending upon thelocation at the image-capturing surface, i.e., different secondimage-capturing conditions are set for different areal portions withinthe second area, will be explained in reference to FIG. 28. FIG. 28presents a schematic diagram illustrating how first image data, andsecond image data are processed.

First image data obtained by capturing an image under the firstimage-capturing conditions, uniformly set for the entire first area atthe image-capturing surface, are output from each of the pixels presentin the first area 141, whereas second image data obtained by capturingan image under the second image-capturing conditions, which are altereddepending upon the location at the image-capturing surface, are outputfrom each of the pixels present in the second area 142. The first imagedata from the first area 141 are output to the first selection unit 151.Likewise, the second image data and second signal data from the secondarea 142 are output to the second selection unit 152.

In the example, uniform first image-capturing conditions are set for theentire first area at the image-capturing surface and accordingly, thefirst selection unit 151 selects the first image data from all thepixels present in the first area. In addition, the secondimage-capturing conditions are altered depending upon the location atthe image-capturing surface. In other words, different secondimage-capturing conditions are set from one areal portion to anotherwithin the second area. The second selection unit 152 selects onlysecond image data generated under specific image-capturing conditionsamong the sets of second image data generated at the individual pixelspresent in the second area but does not select second image datagenerated under other image-capturing conditions. It is to be noted thatsince the first image-capturing conditions and the secondimage-capturing conditions are different from each other, the secondselection unit 152 does not select the second image data as data to beused in the image processing executed for the image data in the firstarea.

In addition, the second selection unit 152 receives the information 181pertaining to the first image-capturing conditions from, for instance,the first selection unit 151.

The generation unit 323 executes image processing such as defectivepixel correction processing, color interpolation processing, edgeenhancement processing and noise reduction processing based upon thefirst image data provided via the first selection unit 151 and outputsimage data having undergone the image processing.

The object detection unit 34 a executes processing so as to detectsubject elements based upon the part of the second image data selectedvia the second selection unit 152, and outputs the detection results.

The setting unit 34 b executes image-capturing condition calculationprocessing such as exposure calculation processing based upon the partof the second image data selected via the second selection unit 152.Then, based upon the calculation results, it divides the image-capturingsurface at the image-capturing unit 32 into a plurality of areascontaining detected subject elements and resets image-capturingconditions for the plurality of areas.

The AF operation unit 34 d executes focus detection processing basedupon the part of the second signal data selected via the secondselection unit 152 and outputs a drive signal, generated based upon thearithmetic operation results, to be used to drive the focus lens in theimage-capturing optical system 31 to the in-focus position.

4. A further example in which the first image-capturing conditions arealtered depending upon the location at the image-capturing surface andthe second image-capturing conditions are also altered depending uponthe location at the image-capturing surface, will be explained inreference to FIG. 29. FIG. 29 presents a schematic diagram illustratinghow first image data, second image data and second signal data areprocessed.

First image data obtained by capturing an image under the firstimage-capturing conditions, which are altered depending upon thelocation at the image-capturing surface, are output from each pixelpresent in the first area 141 and second image data or second signaldata, obtained by capturing an image under the second image-capturingconditions, which are altered depending upon the location at theimage-capturing surface, are output from each pixel present in thesecond area 142. The first image data from the first area 141 are outputto the first selection unit 151. Likewise, the second image data and thesecond signal data from the second area 142 are output to the secondselection unit 152.

As described earlier, the first image-capturing conditions are altereddepending upon the location at the image-capturing surface in thisexample. In other words, different first image-capturing conditions areset for different areal portions within the second area. The firstselection unit 151 selects only first image data obtained under specificimage-capturing conditions, among the sets of first image data obtainedat the individual pixels present in the first area but does not selectfirst image data obtained under other image-capturing conditions. Inaddition, the second image-capturing conditions are altered dependingupon the location at the image-capturing surface. In other words,different second image-capturing conditions are set for different arealportions within the second area. The second selection unit 152 selectsonly second image data obtained under specific image-capturingconditions, among the sets of second image data obtained at theindividual pixels present in the second area but does not select secondimage data obtained under other image-capturing conditions. It is to benoted that since the first image-capturing conditions and the secondimage-capturing conditions are different from each other, the secondselection unit 152 does not select the second image data as data to beused in the image processing executed for the image data in the firstarea.

In addition, the second selection unit 152 receives the information 181pertaining to the first image-capturing conditions from, for instance,the first selection unit 151.

The generation unit 323 executes image processing such as defectivepixel correction processing, color interpolation processing, edgeenhancement processing and noise reduction processing based upon thepart of the first image data selected via the first selection unit 151,and outputs image data having undergone the image processing.

The object detection unit 34 a executes processing so as to detectsubject elements based upon the part of the second image data selectedvia the second selection unit 152 and outputs the detection results.

The setting unit 34 b executes image-capturing condition calculationprocessing such as exposure calculation processing based upon the partof the second image data selected via the second selection unit 152.Then, based upon the calculation results, it divides the image-capturingsurface at the image-capturing unit 32 into a plurality of areascontaining detected subject elements and resets image-capturingconditions for the plurality of areas.

The AF operation unit 34 d executes focus detection processing basedupon the part of the second signal data selected via the secondselection unit 152 and outputs a drive signal, generated based upon thearithmetic operation results, to be used to drive the focus lens in theimage-capturing optical system 31 to the in-focus position.

(Variation 11)

In the third embodiment described above, one selection unit 322corresponds to a single block 111 a (unit block). However, the selectionunits 322 may be each disposed in correspondence to a combined blockmade up with a plurality of blocks 111 a (unit blocks). In such a case,each selection unit 322 sequentially executes data selection processingfor sets of image data output from the pixels belonging to the pluralityof blocks 111 a included in the corresponding combined block. Eventhough the plurality of selection units 322 are each provided incorrespondence to a combined block made up with a plurality of blocks111 a, the plurality of selection units 322 can be engaged in parallelimage data selection processing, which makes it possible to reduce theprocessing onus on the individual selection units 322 and generate anoptimal image in a shorter period of time by using image dataindividually generated in areas with different image-capturingconditions applied thereto.

(Variation 12)

The generation unit 323 in the third embodiment described above isdisposed within the image-capturing unit 32A. As an alternative, thegeneration unit 323 may be disposed outside the image-capturing unit32A. The configuration with an external generation unit 323 disposedoutside the image-capturing unit 32A, too, achieves advantages andoperations similar to the advantages and operations described earlier.

(Variation 13)

The stacked image sensor 100A in the third embodiment described abovefurther includes an image processing chip 114 engaged in the preliminaryprocessing and image processing explained earlier, in addition to theback-side illumination image-capturing chip 111, the signal processingchip 112 and the memory chip 113. However, the present invention may beadopted in a configuration with an image processing unit 32 c disposedat the signal processing chip 112 in place of the image processing chip114 included in the stacked image sensor 100A.

It is to be noted that the various embodiments and variations thereofmay be adopted in any combination.

While various embodiments and variations thereof have been describedabove, the present invention is in no way limited to the particulars ofthese examples. Any mode conceivable within the scope of the technicalteachings of the present invention is also within the scope of thepresent invention.

The devices described below are also part of the embodiments andvariations thereof explained earlier.

(1) An image-capturing device comprising an image sensor that includesan image-capturing area where an image of a subject is captured, asetting unit that sets image-capturing conditions to be applied to theimage-capturing area, a selection unit that selects pixels to be usedfor interpolation among pixels present in the image-capturing area and ageneration unit that generates an image of the subject captured in theimage-capturing area with signals generated through interpolationexecuted by using signals output from the pixels selected by theselection unit. In this image-capturing device, the selection unit makesa change in the selection of at least some of the pixels to be selecteddepending upon the image-capturing conditions set by the setting unit.

(2) An image-capturing device such as that described in (1), in whichthe image sensor includes a first image-capturing area where an image ofa subject is captured and a second image-capturing area where an imageof the subject is captured, the setting unit sets image-capturingconditions for the first image-capturing area and image-capturingconditions for the second image-capturing area, the selection unit makesa change in the selection of at least some of the pixels to be selectedfor purposes of interpolation executed for a pixel present in the firstimage-capturing area, among pixels present in the first image-capturingarea and pixels present in the second image-capturing area, dependingupon the image-capturing conditions set for the second image-capturingarea by the setting unit, and the generation unit generates an image ofthe subject captured in the first image-capturing area with signalsgenerated through interpolation by using signals output from the pixelsselected by the selection unit.

(3) An image-capturing device such as that described in (2), in whichthe selection unit makes a change in the selection of at least some ofthe pixels to be selected for purposes of interpolation executed for apixel present in the first image-capturing area depending upon theimage-capturing conditions set for the first image-capturing area andthe image-capturing conditions set for the second image-capturing areaby the setting unit.

(4) An image-capturing device such as that described in (2) or (3), inwhich the selection unit selects pixels to be used for the interpolationfrom at least either the first image-capturing area or the secondimage-capturing area.

(5) An image-capturing device such as that described in (4), in whichthe selection unit selects a pixel present in the second image-capturingarea as a pixel to be used for the interpolation based upon theimage-capturing conditions set for the second image-capturing area bythe setting unit.

(6) An image-capturing device such as that described in any of (2)through (5), in which if first image-capturing conditions are set forthe first image-capturing area and the second image-capturing area bythe setting unit, the selection unit selects a pixel present in thesecond image-capturing area.

(7) An image-capturing device such as that described in (6), in whichthe selection unit selects a pixel present in the second image-capturingarea as a pixel to be used for the interpolation based upon a valuepertaining to exposure included in the image-capturing conditions setfor the second image-capturing area by the setting unit.

(8) An image-capturing device such as that described in (7), in whichthe selection unit selects a pixel present in the second image-capturingarea as a pixel to be used for the interpolation based upon a valuepertaining to exposure included in the image-capturing conditions setfor the first image-capturing area by the setting unit, and a valuepertaining to exposure included in the image-capturing conditions setfor the second image-capturing area by the setting unit.

(9) An image-capturing device such as that described in (8), in whichthe selection unit selects a pixel present in the second image-capturingarea as a pixel to be used for the interpolation if a difference betweenan exposure value included in the image-capturing conditions set for thefirst image-capturing area by the setting unit and an exposure valueincluded in the image-capturing conditions set for the secondimage-capturing area by the setting unit is equal to or less than 0.3.

(10) An image-capturing device such as that described in any of (2)through (9), in which the selection unit selects pixels present in thefirst image-capturing area if the setting unit sets firstimage-capturing conditions for the first image-capturing area and secondimage-capturing conditions for the second image-capturing area.

(11) An image-capturing device such as that described in (10), in whichthe selection unit selects pixels present in the first image-capturingarea without selecting any pixel present in the second image-capturingarea if the setting unit sets first image-capturing conditions for thefirst image-capturing area and second image-capturing conditions for thesecond image-capturing area.

(12) An image-capturing device such as that described in any of (2)through (11), in which the selection unit selects as a pixel to be usedfor interpolation executed for a first pixel present in the firstimage-capturing area, a third pixel present in the first image-capturingarea set apart from the first pixel by a distance greater than adistance setting apart the first pixel from a second pixel present inthe second image-capturing area.

(13) An image-capturing device such as that described in any of (2)through (12), in which the selection unit selects a varying number ofpixels depending upon the image-capturing conditions set for the secondimage-capturing area by the setting unit.

(14) An image-capturing device such as that described in (13), in whichthe selection unit selects a smaller number of pixels if the settingunit sets first image-capturing conditions for the first image-capturingarea and second image-capturing conditions for the secondimage-capturing area, compared to the number of pixels selected when thefirst image-capturing conditions are set for the first image-capturingarea and the second image-capturing area.

(15) An image-capturing device comprising an image sensor that includesa first image-capturing area set so as to capture an image of a subjectunder first image-capturing conditions, a second image-capturing areaset so as to capture an image of the subject under secondimage-capturing conditions different from the first image-capturingconditions and a third image-capturing area set so as to capture animage of the subject under third image-capturing conditions differentfrom the second image-capturing conditions, a selection unit thatselects pixels to be used for purposes of interpolation executed for apixel present in the first image-capturing area among pixels present inthe second image-capturing area and pixels present in the thirdimage-capturing area, and a generation unit that generates an image ofthe subject captured in the first image-capturing area with signalsgenerated through interpolation executed by using signals output fromthe pixels selected by the selection unit.

(16) An image-capturing device comprising an image sensor that includesa first image-capturing area set so as to capture an image of a subjectunder first image-capturing conditions and a second image-capturing areaset so as to capture an image of the subject under secondimage-capturing conditions different from the first image-capturingconditions, a selection unit that selects pixels to be used for purposesof interpolation executed for a pixel present in the firstimage-capturing area among pixels present in the first image-capturingarea and pixels present in the second image-capturing area, and ageneration unit that generates an image of the subject captured in thefirst image-capturing area with signals generated through interpolationexecuted by using signals output from the pixels selected by theselection unit.

(17) An image-capturing device comprising an image sensor that includesa first image-capturing area where an image of a subject is captured, asecond image-capturing area where an image of the subject is capturedand a third image-capturing area where an image of the subject iscaptured, a setting unit that sets first image-capturing conditions asimage-capturing conditions for the first image-capturing area, setssecond image-capturing conditions, different from the firstimage-capturing conditions, as image-capturing conditions for the secondimage-capturing area and sets third image-capturing conditions,different from the first image-capturing conditions by an extent smallerthan an extent of the difference between the first image-capturingconditions and the second image-capturing conditions, as image-capturingconditions for the third image-capturing area, a selection unit thatselects pixels to be used for purposes of interpolation executed for apixel present in the first image-capturing area among pixels present inthe first image-capturing area, pixels present in the secondimage-capturing area and pixels present in the third image-capturingarea, and a generation unit that generates an image of the subjectcaptured in the first image-capturing area with signals generatedthrough interpolation executed by using signals output from the pixelsselected by the selection unit.

(18) An image-capturing device comprising an image sensor that includesa first image-capturing area where an image of a subject is captured, asecond image-capturing area where an image of the subject is capturedand a third image-capturing area where an image of the subject iscaptured, set apart from the first image-capturing area by a distancegreater than the distance between the first image-capturing area and thesecond image-capturing area, a setting unit that sets image-capturingconditions different from the image-capturing conditions selected forthe first image-capturing area, as image-capturing conditions for thesecond image-capturing area, a selection unit that selects pixels to beused for purposes of interpolation executed for a pixel present in thefirst image-capturing area from pixels present in the firstimage-capturing area, pixels present in the second image-capturing areaand pixels present in the third image-capturing area, and a generationunit that generates an image of the subject captured in the firstimage-capturing area with signals generated through interpolationexecuted by using signals output from the pixels selected by theselection unit.

(19) An image-capturing device comprising an image sensor that includesan image-capturing area where an image of a subject is captured, asetting unit that sets image-capturing conditions to be applied to theimage-capturing area, and a generation unit that generates an image of asubject captured in the image-capturing area with signals generatedthrough interpolation executed by using signals output from pixelsselected as pixels to be used for purposes of interpolation, and presentin the image-capturing area. In this image-capturing device, a change ismade in the selection of at least some of the pixels to be selecteddepending upon the image-capturing conditions set by the setting unit.

(20) An image-capturing device comprising an image sensor that includesa first image-capturing area set so as to capture an image of a subjectunder first image-capturing conditions and a second image-capturing areaset so as to capture an image of the subject under secondimage-capturing conditions different from the first image-capturingconditions, and a generation unit that generates an image of the subjectcaptured in the first image-capturing area with signals generatedthrough interpolation executed by using signals output from pixelsselected as pixels to be used for purposes of interpolation executed fora pixel present in the first image-capturing area among pixels presentin the first image-capturing area and pixels present in the secondimage-capturing area.

(21) An image-capturing device comprising an image sensor that includesan image-capturing area where an image of a subject is captured, asetting unit that sets image-capturing conditions to be applied to theimage-capturing area and a generation unit that generates an image ofthe subject captured in the image-capturing area with signals generatedthrough noise reduction executed by using signals output from pixelsthat output noise reduction signals, selected from pixels present in theimage-capturing area. In the image-capturing device, a change is made inthe selection of at least some of the pixels to be selected dependingupon the image-capturing conditions set by the setting unit.

(22) An image-capturing device comprising an image sensor that includesa first image-capturing area set so as to capture an image of a subjectunder first image-capturing conditions, a second image-capturing areaset so as to capture an image of the subject under secondimage-capturing conditions different from the first image-capturingconditions and a third image-capturing area set so as to capture animage of the subject under third image-capturing conditions differentfrom the second image-capturing conditions, a selection unit thatselects pixels to be used to reduce noise for a pixel present in thefirst image-capturing area among pixels present in the secondimage-capturing area and pixels present in the third image-capturingarea, and a generation unit that generates an image of the subjectcaptured in the first image-capturing area with signals having undergonenoise reduction executed by using signals output from the pixelsselected from the pixels present in the second image-capturing area andthe pixels present in the third image-capturing area as pixels to outputsignals to be used to reduce noise in a signal at a pixel present in thefirst image-capturing area.

(23) An image-capturing device comprising an image sensor that includesa first image-capturing area set so as to capture an image of a subjectunder first image-capturing conditions and a second image-capturing areaset so as to capture an image of the subject under secondimage-capturing conditions different from the first image-capturingconditions and a generation unit that generates an image of the subjectcaptured in the first image-capturing area with signals generatedthrough interpolation executed by using signals output from pixelsselected from pixels present in the first image-capturing area andpixels present in the second image-capturing area, as pixels to outputsignals to be used to reduce noise for a pixel present in the firstimage-capturing area.

(24) An image-capturing device comprising an image sensor that includesan image-capturing area where an image of a subject is captured, asetting unit that sets image-capturing conditions to be applied to theimage-capturing area, and a generation unit that generates an image ofthe subject captured in the image-capturing area with signals havingundergone image processing executed by using signals output from pixelsselected as pixels to be used in image processing. In thisimage-capturing device, a change is made in the selection of at leastsome of the pixels to be selected depending upon the image-capturingconditions set by the setting unit.

(25) An image processing device comprising a selection unit that selectssignals to be used for purposes of interpolation, among signals outputfrom pixels present in an image-capturing area of an image sensor, and ageneration unit that generates an image of a subject captured in theimage-capturing area with signals generated through interpolationexecuted by using the signals selected by the selection unit. Theselection unit in the image processing device makes a change inselection of at least some of the pixels to be selected depending uponthe image-capturing conditions set for the image-capturing area.

(26) An image processing device comprising a selection unit that selectssignals to be used for noise reduction, from signals output from pixelspresent in an image-capturing area of an image sensor, and a generationunit that generates an image of a subject captured in theimage-capturing area with signals having undergone noise reductionexecuted by using the signals selected by the selection unit. Theselection unit in the image processing device makes a change inselection of at least some of the pixels to be selected depending uponthe image-capturing conditions set for the image-capturing area.

(27) An image processing device comprising a selection unit that selectssignals to be used for purposes of interpolation, among signals outputfrom pixels present in an image-capturing area of an image sensor, and adisplay unit that brings up on display an image of a subject captured inthe image-capturing area, generated with signals generated throughinterpolation executed by using the signals selected by the selectionunit. The selection unit in the image processing device makes a changein selection of at least some of the pixels to be selected dependingupon the image-capturing conditions set for the image-capturing area.

(28) An image processing device comprising a selection unit that selectssignals to be used for noise reduction, among signals output from pixelspresent in an image-capturing area of an image sensor, and a displayunit that brings up on display an image of a subject captured in theimage-capturing area, generated with signals having undergone noisereduction executed by using the signals selected by the selection unit.The selection unit in the image processing device makes a change inselection of at least some of the pixels to be selected depending uponthe image-capturing conditions set for the image-capturing area.

Furthermore, the devices described below are also part of theembodiments and variations thereof explained earlier.

(1) An image-capturing device comprising an image sensor that includes afirst area from which first image data obtained by capturing an imageformed with light having entered therein under first image-capturingconditions are output, a second area from which second image dataobtained by capturing an image formed with light having entered thereinunder second image-capturing conditions different from the firstimage-capturing conditions are output, and a third area from which thirdimage data obtained by capturing an image formed with light havingentered therein under the first image-capturing conditions are output,and an image processing unit that generates an image based upon firstimage data resulting from image processing executed by using the secondimage data and at least either the first image data or the third imagedata.

(2) An image-capturing device such as that described in (1), in whichthe image processing unit executes the image processing for first imagedata by using the third image data without using the second image data.

(3) An image-capturing device such as that in (2), in which the firstimage data are output from a pixel at a first position in the firstarea, the second image data are output from a pixel at a second positionin the second area, the third image data are output from a pixel at athird position in the first area, and a distance between the firstposition and the third position is greater than a distance between thefirst position and the second position.

(4) An image-capturing device such as that described in any of (1)through (3), in which the image sensor includes a fourth area from whichfourth image data, obtained by capturing an image formed with lighthaving entered therein under the second image-capturing conditions, areoutput and the image processing unit generates an image based uponsecond image data, resulting from image processing executed by using thefirst image data resulting from the image processing described earlierand at least either the second image data or the fourth image data.

(5) An image-capturing device comprising an image sensor that includes afirst area from which first image data obtained by capturing an imageformed with light having entered therein under first image-capturingconditions are output, and a second area from which second image dataobtained by capturing an image formed with light having entered thereinunder second image-capturing conditions different from the firstimage-capturing conditions are output, and an image processing unit thatgenerates an image based upon first image data resulting from imageprocessing executed by using the first image data and the second imagedata.

(6) An image-capturing device such as that described in (5), in whichthe image processing unit executes the image processing for first imagedata by using the first image data without using the second image data.

(7) An image-capturing device such as that described in (6), in whichthe image processing unit generates an image based upon second imagedata resulting from image processing executed by using the second imagedata, and the first image data resulting from the image processingdescribed above.

(8) An image-capturing device such as that described in any of (1)through (7), in which a plurality of sets of first image data are outputfrom the first area and the image processing unit executes imageprocessing on a set of processing target first image data, among theplurality of sets of first image data, by using another set of firstimage data among the plurality of sets of first image data.

(9) An image-capturing device such as that described in (8), in whichthe first image data are output from a pixel at a first position in thefirst area, the second image data are output from a pixel at a secondposition in the second area, the other set of first image data is outputfrom a pixel at a third position in the first area, and the distancebetween the first position and the third position is greater than thedistance between the first position and the second position.

(10) An image-capturing device such as that described in any of (1)through (9), in which the first image-capturing conditions and thesecond image-capturing conditions both include at least either anaccumulation time setting or an ISO sensitivity setting.

(11) An image-capturing device such as that described in any of (1)through (10), in which the image processing unit processes the firstimage data by executing at least defective pixel correction processing,color interpolation processing, edge enhancement processing or noisereduction processing.

(12) An image-capturing device such as that described in any of (1)through (11), in which the image processing unit includes a selectionunit that selects image data to be used in image processing executed onthe first image data.

(13) An image-capturing device such as that described in (12), in whichthe image sensor is capable of executing image-capturing operation bysetting different image-capturing conditions from one unit area toanother at an image-capturing surface, and the selection unit, disposedin correspondence to each unit area or each combined area made up with aplurality of unit areas, selects image data from the corresponding unitarea or from the unit areas present in the corresponding combined area.

(14) An image-capturing device such as that described in (13), in whichthe image sensor is disposed at a first semiconductor substrate and theselection unit is disposed at a second semiconductor substrate.

(15) An image-capturing device such as that described in (14), in whichthe first semiconductor substrate and the second semiconductor substrateare stacked one upon the other.

(16) A display device having a display unit that brings up on display animage generated based upon first image data output by capturing an imageformed with light having entered a first area at an image-capturing unitunder first image-capturing conditions, second image data output bycapturing an image formed with light having entered a second area thatthe image-capturing unit, under second image-capturing conditionsdifferent from the first image-capturing conditions, and third imagedata output by capturing an image formed with light having entered athird area at the image-capturing unit under the first image-capturingconditions. The display device further includes an image processing unitthat generates an image based upon first image data, resulting fromimage processing executed by using at least either the first image dataor the third image data, and the second image data, and the display unitbrings up on display the image generated by the image processing unit.

(17) A display device comprising an image processing unit that generatesan image based upon first image data resulting from image processingexecuted by using first image data output from a first area at animage-capturing unit where an image formed with light having enteredtherein is captured under first image-capturing conditions and secondimage data output by capturing an image formed with light having entereda second area at the image-capturing unit under second image-capturingconditions different from the first image-capturing conditions, and adisplay unit that brings up on display the image generated by the imageprocessing unit.

(18) An image processing device having an image processing unit thatgenerates an image based upon first image data output by capturing animage formed with light having entered a first area at animage-capturing unit under first image-capturing conditions, secondimage data output by capturing an image formed with light having entereda second area at the image-capturing unit under second image-capturingconditions different from the first image-capturing conditions, andthird image data output by capturing an image formed with light havingentered a third area at the image-capturing unit under the firstimage-capturing conditions. The image processing unit generates theimage based upon first image data, resulting from image processingexecuted by using at least either the first image data or the thirdimage data, and the second image data.

(19) An image processing device having an image processing unit thatgenerates an image based upon first image data resulting from imageprocessing executed by using first image data output from a first areaat an image-capturing unit where an image formed with light havingentered therein is captured under first image-capturing conditions, andsecond image data output by capturing an image formed with light havingentered a second area at the image-capturing unit under secondimage-capturing conditions different from the first image-capturingconditions.

The disclosure of the following priority application is hereinincorporated by reference:

Japanese Patent Application No. 2015-195288 filed Sep. 30, 2015

REFERENCE SIGNS LIST

-   1, 1C . . . camera-   1B . . . image-capturing system-   32 . . . image-capturing unit-   32 a, 100 . . . image sensor-   33 . . . image processing unit-   33 a, 321 . . . input unit-   33 b, 322 . . . selection unit-   33 c, 323 . . . generation unit-   34 . . . control unit-   34 a . . . object detection unit-   34 b . . . area dividing unit-   34 d . . . image-capturing control unit-   35 . . . display unit-   90 . . . predetermined range-   1001 . . . image-capturing device-   1002 . . . display device-   P . . . target pixel

The invention claimed is:
 1. An image-capturing device, comprising: animage sensor that includes a first image-capturing area set so as tocapture an image of a subject under first image-capturing conditions, asecond image-capturing area set so as to capture an image of the subjectunder second image-capturing conditions different from the firstimage-capturing conditions and a third image-capturing area set so as tocapture an image of the subject under third image-capturing conditionsdifferent from the second image-capturing conditions; and a selectionunit that selects pixels, to be used for purposes of interpolationexecuted for a pixel present in the first image-capturing area amongpixels present in the second image-capturing area, and pixels present inthe third image-capturing area.
 2. The image-capturing device accordingto claim 1, further comprising: a generation unit that generates animage of the subject captured in the first image-capturing area withsignals generated through interpolation executed by using signals outputfrom the pixels selected by the selection unit.
 3. The image-capturingdevice according to claim 1, further comprising: a detection unit thatdetects the subject captured in the first image-capturing area withsignals generated through interpolation executed by using signals outputfrom the pixels selected by the selection unit.
 4. The image-capturingdevice according to claim 1, further comprising: a control unit thatcontrols setting of image-capturing conditions set in the firstimage-capturing area with signals generated through interpolationexecuted by using signals output from the pixels selected by theselection unit.
 5. The image-capturing device according to claim 1,further comprising: the first image-capturing area, the secondimage-capturing area and the third image-capturing area being set so asto capture an image of the subject through a lens that adjusts anin-focus position of an optical system; and a control unit that controlsdriving of the lens based upon a state of the in-focus position of theoptical system with signals generated through interpolation executed byusing signals output from the pixels selected by the selection unit. 6.The image-capturing device according to claim 1, wherein: the thirdimage-capturing conditions which are to be different from the firstimage-capturing conditions by an extent smaller than an extent ofdifference between the first image-capturing conditions and the secondimage-capturing conditions, are set as image-capturing conditions forthe third image-capturing area.
 7. The image-capturing device accordingto claim 6, wherein: the first image-capturing conditions are set forthe third image-capturing area.
 8. The image-capturing device accordingto claim 1, wherein: the third image-capturing area is set apart fromthe first image-capturing area by a distance greater than a distancebetween the first image-capturing area and the second image-capturingarea.